Gatsby DSG Build on Netlify fails due to function size
opened this issue · comments
My gatsby build on netlify fails due to the size of the DSG/SSR functions which are created by netlify-plugin-gatsby. Error produced is Request must be smaller than 69905067 bytes for the CreateFunction operation.
Additionally, during the 'onPostbuild' section of the build, I get the following message:
The function zip ../../../tmp/zisi-6212bb95da69157a8c04d77d/__ssr.zip size is 266 MB, which is larger than the maximum supported size of 52.4 MB.
I'm able to get my build to function locally but when I serve the site, the DSG pages (which get successfully built as specified in build output log) don't work and return 404's.
Build is running @netlify/plugin-gatsby 2.0.2 & gatsby 4.5.3.
I've contacted Netlify support, and they suggested this is to do with the functions from this plugin and not Netlify itself - they're hitting AWS's size limitation.
Any help would be greatly appreciated!
Hey! The build logs should include details of what are the largest files in the zip. Can you share the logs?
This is related to ticket https://netlify.zendesk.com/agent/tickets/84013 from an Enterprise customer in our helpdesk, where there are things like:
- a link to build logs
- a link to our logging service showing that this happens occasionally to several folks but that several is literally 10 other customers out of millions.
- a mention that the same code deploys on gatsby's cloud a-ok despite the very large file that we seem to create
Thanks in advance for your help, @ascorbic !
Thanks, @fool. The file mentioned in the ticket (data.mdb) is the Gatsby data layer database. @scottgimblettnz This is created by Gatsby during the build process, and is required. That is a very large database though. How many pages does your site have? If you run gatsby build --verbose, it should also log the number of created nodes and pages. Which Gatsby source plugins are you using? You may be able to cut this down by restricting the amount of data being loaded, which is something lots of source plugins can do. This would also improve build times.
OK, I've realised I have access to your logs so am able to diagnose some of it. You appear to have several 150MB(!) json files in there, which are static query results. If you load the current index page for the site, you will see in the network panel that this is being downloaded for every page. This kind of thing happens when you are making static queries in Gatsby that return extremely large results. Are there static queries that are returning every page in the data store?
@ascorbic Thanks for the response. It appears we do have a static query which returns every page, yes. This particular build has multiple languages available, and in some cases is building a version of a page for each language we have. If my build output is accurate, we're building something like 15,000+ pages (hence the desire to use DSG). I suppose this would explain why the json files are so large!
I'm having a similar issue on my site, where I'm seeing mp4 files being bundled in all of my function zip files, causing them all to exceed the maximum function size. Deploy log
I ended up working around this by adding the following to my netlify.toml:
[functions]
included_files = ["!.cache/page-ssr/routes/static/*.mp4"]It seems like the entire .cache directory is being bundled with each function. Is that typical?
@trevorblades Not the whole .cache directory, but the page-ssr directory is, because it may be needed for SSR
I just ran up against this issue again for another site. Functions had been bundled and uploaded without issue for a long time, and suddenly they started failing because the function size was >50 MB. I added a line that excludes .cache/page-ssr/routes/static/** and they started working again.
However, this has me worried that as we continue adding pages and more content to the site, our functions will eventually become too large to upload, and there won't be any other globs that I can reasonably exclude to make them work again.
At that point, we'd be stuck. Reducing the amount of static content (MDX files rendered statically) isn't an option. It seems odd that all of those statically rendered MDX pages would be needed for our SSR pages, but I don't know what files I should or shouldn't exclude from the function bundles.
I ran into this issue for a Gatsby site that uses Contentful as a source.
I fixed it by using a filter to reduce the function size by about 70% to fit under a 50 MB limit.
Inside gatsby-config.js,
{
resolve: "gatsby-source-contentful",
options: {
localeFilter: locale => locale.code === "en-US",
},
},
We are having the same issue:
https://app.netlify.com/sites/aiid/deploys/62d1ab99dec9430009a8a6a9
This is the project:
https://github.com/responsible-ai-collaborative/aiid
Looks like bundling the .cache folder will always end up in this issue as soon as the site grows enough. For example, this project is very text heavy, and there is not much that could be removed.
I'm having a similar issue with my __ssr.zip file being too large.
We implemented SSR on the careers section of our site and ran into the same error when we deployed it to Netlify: The function zip ../<path>/__ssr.zip size is 108 MB, which is larger than the maximum supported size of 52.4 MB.
The __ssr.zip file includes all pages and templates in the .cache/page-ssr/routes directory (even the ones that didn't implement SSR), which pushes it over the limit. I added the following settings to our netlify.toml file to exclude all files except the ones with SSR.
// netlify.toml
[functions]
included_files = ["!.cache/page-ssr/routes/component---src-pages-!(careers)*.js", "!.cache/page-ssr/routes/component---templates-*.js"]
This fixed the size issue and seems to be working in our deploy preview, but I'm nervous about pushing it into production. I'm assuming the pages that aren't using SSR don't need to be in __ssr.zip, but I don't know for sure. I haven't dug deep enough to understand it. I also don't like the idea of modifying the neltify.toml file every time I implement SSR on another area of the site. I can see it causing issues down the road.
If that's the fix, could the plugin filter out files that don't export getServerData? Looks like it would have to modify the included_files to pass to zip-it-and-ship-it, but I'm sure it could be done.
The site I'm referring to is named hutson. Deploy id with the error is 636a73d3975e8d009cad5a66 and the working deploy after changing the included_files in netlify.toml is 63729fed93fdf800097e1e3b. Case number 117675. Thanks!
@AustinLeeGordon your fix should be fine. Yes, it's a good idea to try to filter them like that. I will open an issue. The trouble is that Gatsby doesn't provide that information as part of the build, so we'd need to extract it in gatsby-plugin-netlify and write it out so we can use it later.
@AustinLeeGordon your fix should be fine. Yes, it's a good idea to try to filter them like that. I will open an issue. The trouble is that Gatsby doesn't provide that information as part of the build, so we'd need to extract it in
gatsby-plugin-netlifyand write it out so we can use it later.
@ascorbic Sounds great! Thank you for the quick response!
What is the recourse for using Gatsby's useStaticQuery + DSG on Netlify?
I have many components in Gatsby that need localized data fetching, hence the need for useStaticQuery. Since static queries can't take variables, they do fetch large data sets which appear to bloat the .cache directory and data.json files.
I've already set the env var so my data.mdb gets hosted on a CDN but my __dsg.zip is still 115mb. I have many components in .cache/page-ssr/sq-content directory that are 19mb+.
Gatsby Cloud is building with DSG enabled just fine which pains me because I don't want that out as a solution to this problem.
Those size problems mentioned here are better solved in gatsby directly.
I started gatsbyjs/gatsby#37713 for what improvement could be made there. Some improvements already did happen - the .cache/page-ssr/routes/static/ problem specifically was handled ( gatsbyjs/gatsby#37284 ).
I did prepare 2 ideas to tackle sq-content and .cache/page-ssr/routes directories getting huge (mostly about gatsby doing some deduplication work to prevent copies of same thing)
Having to move from the dying Gatsby Cloud to Netlify followed all steps, but the build fails on Netlify with:
The function "__DSG" is larger than the 50MB limit. Please consider reducing it.
Note that all "defer" values are set to "false" in gatsby-node.js
Is there a quick fix?
(The Netlify support will perhaps get back in 4 days, unclear if they will fix it. Basically, right at the time Gatsby Cloud will shut down.)
@Vacilando Same issue. One of my client websites has ~ 130k pages, of which only 16% is done as SSG. This works fine on Gatsby Cloud, but the DSG function on Netlify just doesn't work for them. Concerned as deadline is looming - only a month to go. Frontend team is currently trying to upgrade from v4 to v5 to see if that resolves some issues, but an alternative is perhaps to switch to SSR, but that largely negates the benefits of Gatsby over what we had previously.
That's assuming that SSR doesn't fall foul of the same issue that plagues DSG
This is so sad. After Gatsby Cloud was sold to Netlify, Netlify should have made 100% sure their parameters match those of Gatsby Cloud, so that the migrations are painless. Because like this many of us are hostages of the situation — Gatsby Cloud perishing, Netlify hosting not working out of the box, people like us having to make decisions about deep analysis or even redesign of the apps. Honestly, if I am forced to redesign I may as well do it in NextJS.
One of my client websites has ~ 130k pages, of which only 16% is done as SSG. This works fine on Gatsby Cloud, but the DSG function on Netlify just doesn't work for them. Concerned as deadline is looming - only a month to go. Frontend team is currently trying to upgrade from v4 to v5 to see if that resolves some issues, but an alternative is perhaps to switch to SSR, but that largely negates the benefits of Gatsby over what we had previously.
That's assuming that SSR doesn't fall foul of the same issue that plagues DSG
@Antaris, did the rework to SSR solve the problem for you?