The DSG function deploys fail with a "larger than the maximum supported" error for high page counts.
overlordofmu opened this issue · comments
Description of the issue
The DSG feature is handled by Functions that are created by this plugin. However, the plugin doesn't detect when the database files used will cause the deployed Function will exceed the limits at AWS:
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
I believe MDB flat database files are used in these functions. (I could be wrong about this.) As these become too large the function size exceeds the Lambda limits and deploys will always fail.
Reproduction
Enable DSG for a large number of pages and then attempt to build and deploy at Netlify
The error in the build logs will be similar to this:
The function zip ../../../tmp/zisi-<hash>/__dsg.zip size is 199.9 MB, which is larger than the maximum supported size of 52.4 MB.
Expected behavior
Ideally, the plugin would detect this limit and create multiple functions and redirect rules. The redirect would send each DSG URL to the function with the database which can generate the page. It would automatically split the MDB files into many smaller files and create multiple functions. Each function would handle a subset of the total pages to keep the bundle size under the Lambda limit.
I realize that is a non-trivial feature request. However, besides creating more than one function, I can see no other solution for this plugin to automatically scale to an arbitrary large number of DSG pages. If you use a single function at some point the MDB file will exceed the limit.
Hey. This needs solving in the build plugin rather than here, so I'm going to move the issue. Splitting by route is unfortunately impossible, because the datastore includes the whole of the Gatsby data layer and we can't track the data dependency. I have a possible solution here: #320
#320 has been merged. Can you try the latest plugin @overlordofmu? See https://docs.netlify.com/integrations/frameworks/gatsby/#optional-loading-of-the-gatsby-datastore-from-the-cdn
Closing due to inactivity. If it is still an issue @overlordofmu , please re-open.