Netlify DSG page failed with 500 server error when using custom source plugin.
MuhammadHadiofficial opened this issue · comments
Describe the bug
Hi, we are upgrading to gatsbyv4 and using netlify to host our site. While adding DSG support for several pages. Unexpected behavior was seen.
When the site is deployed for the very first time.
DSG pages give 500 internal errors on DSG enabled pages.
When I redeploy without clearing cache these pages automatically work.
These are the logs for the _dsg function.
Feb 13, 10:21:00 PM: ebd60328 error Error: ENOENT: no such file or directory, open '/opt/build/repo/.cache/caches/gatsby-source-custom-api/44b5d8b9f62a9d1bf85708e10185900a/EoHoW6.jpeg' Feb 13, 10:21:00 PM: ebd60328 at Object.openSync (fs.js:497:3) Feb 13, 10:21:00 PM: ebd60328 at Object.readFileSync (fs.js:393:35)Feb 13, 10:21:00 PM: ebd60328 at Object.lfs.<computed> [as readFileSync] (/var/task/.netlify/functions-internal/__dsg/__dsg.js:3284:24)Feb 13, 10:21:00 PM: ebd60328 at Function.hasha.fromFileSync (/var/task/.cache/query-engine/index.js:175592:54) Feb 13, 10:21:00 PM: ebd60328 at createFileHash (/var/task/.cache/query-engine/index.js:175130:25) Feb 13, 10:21:00 PM: ebd60328 at /var/task/.cache/query-engine/index.js:175283:22 Feb 13, 10:21:00 PM: ebd60328 at Array.map (<anonymous>) Feb 13, 10:21:00 PM: ebd60328 at createInternalJob (/var/task/.cache/query-engine/index.js:175280:50) Feb 13, 10:21:00 PM: ebd60328 at /var/task/.cache/query-engine/index.js:177000:23Feb 13, 10:21:00 PM: ebd60328 at /var/task/.cache/query-engine/index.js:52897:18Feb 13, 10:21:00 PM: ebd60328 at /var/task/.cache/query-engine/index.js:26666:12Feb 13, 10:21:00 PM: ebd60328 at Object.doubleBoundActionCreators.<computed> [as createJobV2] (/var/task/.cache/query-engine/index.js:160607:20) Feb 13, 10:21:00 PM: ebd60328 at createJob (/var/task/.cache/query-engine/index.js:186952:27)Feb 13, 10:21:00 PM: ebd60328 at batchQueueImageResizing (/var/task/.cache/query-engine/index.js:187044:27) Feb 13, 10:21:00 PM: ebd60328 at generateImageData (/var/task/.cache/query-engine/index.js:193095:48) Feb 13, 10:21:00 PM: ebd60328 at resolve (/var/task/.cache/query-engine/index.js:292538:25) Feb 13, 10:21:00 PM: ebd60328 Duration: 100.57 ms Memory Usage: 578 MB
It's very unclear to me why this is happening. gatsby-plugin-image is supposed to cache all the images automatically.
Or
the reason is because netlify-plugin-gatsby still doesn't support images as mentioned that staticImage and some others are still not supported?
Sidenote:
We are using aws s3 for hosting images.
Expected Behavior:
If the DSG pages are being rendered properly on second deploy then they are supposed to be working on first ever deployment also.
Screenshots
Link to URL with DSG enabled:
https://62012bfbdcbc1d0008521095--tender-galileo-fdd375.netlify.app/products/1200-peak-amp-portable-car-jump-starterportable-power-station-with-120-psi-compressor-and-500-watt-p/
This URL might be working right now for you because I've already redeployed the site.
HI @MuhammadHadiofficial ,
This is likely to be because we don't bundle images inside the DSG functions by default, so they can't find them. This is because they can make the function bundle larger than the 50MB supported by AWS Lambda. This is also why we say that we don't support sharp and StaticImage in DSG pages. Could you explain what you are doing with these images? If there aren't too many of them, you could in theory add them to the bundle by adding this to your netlify.toml:
[functions]
included_files = [".cache/caches/gatsby-source-custom-api/**/*"]Let me know if this helps.
HI @MuhammadHadiofficial ,
This is likely to be because we don't bundle images inside the DSG functions by default, so they can't find them. This is because they can make the function bundle larger than the 50MB supported by AWS Lambda. This is also why we say that we don't support sharp and StaticImage in DSG pages. Could you explain what you are doing with these images? If there aren't too many of them, you could in theory add them to the bundle by adding this to yournetlify.toml:[functions] included_files = [".cache/caches/gatsby-source-custom-api/**/*"]Let me know if this helps.
Hi @ascorbic, I'm afraid adding them to netlify.toml won't help because that is an ecommerce site and for each product there are more then 4 to 5 images. So, if I have 400 products 4images x 400 products = 1600 images the bundle for aws lambda would explode.
Is there any other workaround this problem as bundle is going to be limited to 50mb always.
Should I use CDNs or any potential solution because in our use case for ecommerce app DSG will definitely reduce our build time at Netlify as it is must have.
Any help?
Understood. What's the data source for the images? Are you using a CMS, Shopify etc? Many of them support serving the images directly, and there are ways to do this in Gatsby Image.
We have our custom source plugin attached to our own cms. Plugin receives image links from S3.
Oh yes. A great option would be an image CDN such as Imgix or Cloudinary. They allow you to use the S3 bucket as a source, and would mean there would be no build time or downloads needed. Imgix has a Gatsby plugin designed for exactly this too, which works with the Gatsby Image plugin.
Thanks, @ascorbic for the direction. I explored imgix but couldn't understand at first how it works. I'll give it one more go and explore the option that you mentioned " they allow s3 bucket as a source" and try to form a solution.
Thanks again, I'll reach back to you if I'm stuck again.
Happy to help!
Hi @MuhammadHadiofficial, just wondering if you were able to get things working?
Hi,@nickytonline we moved to some other priority items and didn't test that again but I believe with new Gatsby upgrade it might work. As new upgrade already resolved alot of our issues lately.
Sounds good @MuhammadHadiofficial. I’m going to go ahead and close this for now but if you’re still having the issue. Just let us know.