antfu / vite-ssg

Static site generation for Vue 3 on Vite

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Large sites and / or sites with large initialState can crash Node due to memory exhaustion

gryphonmyers opened this issue · comments

Just leaving a note of this issue here. I will be submitting some PRs soon to address.

  • Add concurrency option (this alleviates the majority of the issue)
  • Consider rendering initialState into the HTML document AFTER HTML has been run through jsdom and minifier. I think large initialState being in the document during these steps is slowing them down, and I can't think of a great reason why it needs to be there.
  • Put render jobs into worker threads

I'm seeing this too and curious if there's some sort of memory leak going on? Wondering why memory used by node increases over the render process..

I'm seeing this too and curious if there's some sort of memory leak going on? Wondering why memory used by node increases over the render process..

I don't think it's a memory leak. It's that you get many pages building in parallel and each page's initial state payload takes a bite out of the overall memory budget. Eventually if there are pages that use enough memory during render, this can lead to exhaustion.

You can work around this by setting concurrency to a lower number. The concurrency option did get added, and this solves the majority of the problem. The other items on my list here, which I obvs never did, involve making each page eat less memory during render, basically. I don't think it's just that initial state that is eating up budget - there are additional processes that use more memory and I think those can be optimized.

Hmm.. what I'm seeing is even with concurrency set to something very low, like 5, node's memory usage grows and grows over the course of the build process until it ultimately crashes. Not sure if that could be a memory leak in my application code?

Ah! Hmm, that's not something I ever encountered. For me, setting concurrency completely resolved the OOM issues I was getting. That doesn't necessarily mean that there is definitely not a memory leak in the vite-ssg code, but I would suggest that you re-examine your source to see if there is anywhere you might be accumulating memory footprint and not realizing it.