Issue with downloaded images when running chapter 2 code
mattwelke opened this issue · comments
I'm in stage 2 of the Manning review right now, and I wanted to rerun chapter 2 to refresh my memory. I'm starting with the code in this repo as of July 28th. I was able to deploy the services for the distributor, crawler, and analysis, and it looks like they're being invoked properly, but the crawler service isn't getting the images properly. The bucket ends up with a few files with UUID names and no extension. Each file is 37.0 B in size. The status.json after the analysis service finishes shows errors coming back from Rekognition because it isn't able to use them because there's no recognized file format.
My status.json after the test message in SQS before deploying the analysis service:
{
"url": "https://www.fourtheorem.com",
"stat": "downloaded",
"downloadResults": [
{
"url": "https://user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_630,w_1200,f_auto,q_auto/1683166/fourTheorem-Icon-Colour_ffdaqr.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1887969/336615_754571.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/803591_493588.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/627290_748283.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/991972_577611.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/262334_782130.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1683166/4TheoremHR-206c_atmqag.jpg",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1683166/IMG_2607_s3mgsc.jpg",
"stat": null
}
]
}My status.json after deploying the analysis service and sending the second message to the SQS queue:
{
"url": "https://www.fourtheorem.com",
"stat": "analyzed",
"downloadResults": [
{
"url": "https://user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_630,w_1200,f_auto,q_auto/1683166/fourTheorem-Icon-Colour_ffdaqr.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_300,w_300,f_auto,q_auto/1683166/464788_60136.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1887969/336615_754571.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/803591_493588.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/627290_748283.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/991972_577611.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1305285/262334_782130.png",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1683166/4TheoremHR-206c_atmqag.jpg",
"stat": null
},
{
"url": "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"stat": "ok"
},
{
"url": "https://www.fourtheorem.com//user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/1683166/IMG_2607_s3mgsc.jpg",
"stat": null
}
],
"analysisResults": [
{
"image": "www.fourtheorem.com/145e4580-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.559Z",
"requestId": "f9f3299c-b185-11e9-911a-5f092d135c3f",
"statusCode": 400,
"retryable": false,
"retryDelay": 0.6624038143925759
}
},
{
"image": "www.fourtheorem.com/146152c0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.557Z",
"requestId": "f9f377be-b185-11e9-b7db-31d96e79290b",
"statusCode": 400,
"retryable": false,
"retryDelay": 40.707824820991
}
},
{
"image": "www.fourtheorem.com/1461a0e0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.560Z",
"requestId": "f9f37794-b185-11e9-ae21-030afa8ccd80",
"statusCode": 400,
"retryable": false,
"retryDelay": 72.60318403685527
}
},
{
"image": "www.fourtheorem.com/1461c7f0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.579Z",
"requestId": "fa03081c-b185-11e9-b458-61c9b4f3c044",
"statusCode": 400,
"retryable": false,
"retryDelay": 75.50197600936184
}
},
{
"image": "www.fourtheorem.com/1461ef00-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.578Z",
"requestId": "fa02e0eb-b185-11e9-99cf-fde9012fdaea",
"statusCode": 400,
"retryable": false,
"retryDelay": 89.12434889307994
}
},
{
"image": "www.fourtheorem.com/14621610-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.559Z",
"requestId": "f9f6d338-b185-11e9-950c-a56b94f0b5c8",
"statusCode": 400,
"retryable": false,
"retryDelay": 7.097429440553493
}
},
{
"image": "www.fourtheorem.com/146438f0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.578Z",
"requestId": "fa02e080-b185-11e9-8f5f-c1e9f6e2144c",
"statusCode": 400,
"retryable": false,
"retryDelay": 54.803948292451764
}
},
{
"image": "www.fourtheorem.com/f900b290-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.560Z",
"requestId": "f9f6fa10-b185-11e9-a239-f7c0997dcd57",
"statusCode": 400,
"retryable": false,
"retryDelay": 88.11870915598827
}
},
{
"image": "www.fourtheorem.com/f90100b0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.560Z",
"requestId": "f9f6ac27-b185-11e9-92d4-2f7f37e9347d",
"statusCode": 400,
"retryable": false,
"retryDelay": 9.06639965443401
}
},
{
"image": "www.fourtheorem.com/f9014ed0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.581Z",
"requestId": "fa030802-b185-11e9-8df6-c5fa97e5dca4",
"statusCode": 400,
"retryable": false,
"retryDelay": 87.23615918185952
}
},
{
"image": "www.fourtheorem.com/f902ae60-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.579Z",
"requestId": "fa0355bb-b185-11e9-996c-17153a389d8b",
"statusCode": 400,
"retryable": false,
"retryDelay": 0.9124073598199667
}
},
{
"image": "www.fourtheorem.com/f902d570-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.579Z",
"requestId": "fa0307dc-b185-11e9-ba7d-bbb63ed177b0",
"statusCode": 400,
"retryable": false,
"retryDelay": 43.74453268373555
}
},
{
"image": "www.fourtheorem.com/f90609c0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.578Z",
"requestId": "fa0355ab-b185-11e9-8f76-b17b3ef1cf31",
"statusCode": 400,
"retryable": false,
"retryDelay": 20.091979676529604
}
},
{
"image": "www.fourtheorem.com/f90630d0-b185-11e9-a028-d73a3911036d",
"labels": [],
"err": {
"message": "Request has invalid image format",
"code": "InvalidImageFormatException",
"time": "2019-07-28T22:20:59.580Z",
"requestId": "fa032eed-b185-11e9-85ed-1d87e10b7330",
"statusCode": 400,
"retryable": false,
"retryDelay": 94.550262322752
}
}
],
"wordCloudList": []
}Some system info:
> aws --version
aws-cli/1.16.148 Python/3.6.7 Linux/5.0.0-21-generic botocore/1.12.138
> sls --version
1.48.4 (Enterprise Plugin: 1.3.3, Platform SDK: 2.1.0)
> node --version
v8.16.0
> npm --version
6.10.2
Thanks for reporting this! The issues is that the site fourTherorem.com has been changed recently and the CMS being used is now displaying images in WebP format. Rekognition only supports jpeg and png. We will update the text to use a different URL. In the meantime you should be able to point the chapter 2 system at an alternative URL and see some better results..
Gotcha, will try it out and then continue with the rest of the chapters.
Thanks @Welkie . We have corrected the manuscript and you should have an updated chapter 2 with you shortly. Appreciate the feedback!
@pelger Chapter 2 worked great this time! 👍