hexo generate slow with 400 posts

Question

hexo generate slow with 400 posts

iMilnb opened this issue 8 years ago · comments

$ hexo version
hexo: 3.1.1
os: NetBSD 7.0_STABLE netbsd x64
http_parser: 2.5.0
node: 4.1.1
v8: 4.5.103.33
uv: 1.7.4
zlib: 1.2.3
ares: 1.10.1-DEV
modules: 46
openssl: 1.0.2d
$ hexo clean
INFO  Deleted database.
INFO  Deleted public folder.
$ hexo g
[...]
INFO  1071 files generated in 3.68 min
# without cleaning
$ hexo g
INFO  Files loaded in 4.14 s
INFO  0 files generated in 4 min

Is this a normal generation time for 400 posts? seems awfully slow to me. Every change I make on a layout or a css takes ages to be generated. hexo g -w is barely unusable.

FWIW:

highlight:
  enable: true
  line_number: true
  auto_detect: false
  tab_replace:

Similar results with enable: false.

Theme is light.

leesei · Answer 1 · Mon Feb 22 2016 15:02:04 GMT+0800 (China Standard Time)

Did you tried hexo server?

iMil · Answer 2 · Mon Feb 22 2016 15:34:43 GMT+0800 (China Standard Time)

I did, it is indeed more usable and faster, but only for pre-visualization / debugging purposes

leesei · Answer 3 · Mon Feb 22 2016 21:58:08 GMT+0800 (China Standard Time)

I see. I was trying to suggest you to use it during development, so that generate is used sparingly.
What about not cleaning before every generate? It should be able to pick up changes in posts (a theme change will need to be cleaned though).

iMil · Answer 4 · Mon Feb 22 2016 22:06:50 GMT+0800 (China Standard Time)

Sure, I don't run clean everytime, just for testing, but TBH, even hexo g without a previous clean takes ages...

$ hexo g
INFO  Files loaded in 4.14 s
INFO  0 files generated in 4 min

iMil · Answer 5 · Tue Feb 23 2016 22:51:02 GMT+0800 (China Standard Time)

For the sake of it, I've tried to hexo g on a tmpfs mounted directory, and the result is pretty much the same. Meaning it's not about filesystem access but really processing time.

Xuanwo · Answer 6 · Thu Feb 25 2016 20:30:48 GMT+0800 (China Standard Time)

I don't think it's normal.
As in my blog: https://travis-ci.org/Xuanwo/xuanwo.github.io/builds/111696580 ,

INFO  Files loaded in 4.46 s
INFO  718 files generated in 22 s

The curve of time/files is strange from 22/718(about 0.03) to 240/1071(about 0.224).

Maybe you can try set line_number to false to disable highlight.js totally.

iMil · Answer 7 · Thu Feb 25 2016 23:53:44 GMT+0800 (China Standard Time)

Well:

highlight:
  enable: false
  line_number: false
  auto_detect: false
  tab_replace:

$ hexo g 
INFO  Files loaded in 4.35 s
[...]
INFO  36 files generated in 4.08 min

Doesn't seem to run faster. Any hint on how to help debug this?

leesei · Answer 8 · Fri Feb 26 2016 00:18:06 GMT+0800 (China Standard Time)

Using genstubs to generate 500 posts: ./genstubs.js -N500 -P20

Symlinking source to https://github.com/leesei/hexo-vanilla-site

hexo clean
hexo g --debug

00:10:36.697 DEBUG Hexo version: 3.1.1
...
00:10:38.561 INFO  Files loaded in 1.7 s
...
00:10:42.014 INFO  723 files generated in 3.45 s

(Disclaimer: I'm on SSD, but since you've tried with tmpfs, it shouldn't matter)
Here's the full log: hexo-stubs.txt

Please check where most of the time is spent on your run.

iMil · Answer 9 · Fri Feb 26 2016 21:05:05 GMT+0800 (China Standard Time)

--debug seems to indicate the rendering is the slowest process, ~0.3s by post. Is there any way to dig deeper on the rendering process to know which part is the slowest?

leesei · Answer 10 · Fri Feb 26 2016 22:58:56 GMT+0800 (China Standard Time)

@tl3shi I cloned your repo
It took 16mins to run hexo g --debug, however if I changed the theme from maupassant to landscape, it took 16 seconds.
maupassant used 0.6s to render a post vs 0.1s to that of landsacape. Maybe you can reflect this to the author.

@iMilnb what theme are you using? Can you give landscape a try and report.

iMil · Answer 11 · Sat Feb 27 2016 02:35:33 GMT+0800 (China Standard Time)

theme: landscape

INFO  1074 files generated in 15 s

Boom! that's it, it took only 15 seconds to render the 400 posts with landscape

leesei · Answer 12 · Sat Feb 27 2016 02:37:32 GMT+0800 (China Standard Time)

Alas, though this is not a Hexo problem, it seems easy to write an inefficient theme.

iMil · Answer 13 · Sat Feb 27 2016 05:21:51 GMT+0800 (China Standard Time)

Sad thing is that I used the light theme which has been forked by many themes and so is spreading around a lot; so the question remains, how to dig the debug deeper to understand what makes the rendering so slow in comparison to landscape. Meanwhile I'll open an issue at https://github.com/hexojs/hexo-theme-light

iMil · Answer 14 · Sat Feb 27 2016 15:50:58 GMT+0800 (China Standard Time)

Ok I found that it's the tagcloud widget that slows the pages generation. Without it I get:

INFO  840 files generated in 3.92 s

Even faster than landscape.

Edit

The tag widget also slows down page generation. FWIW we're talking about 276 tags.

leesei · Answer 15 · Sat Feb 27 2016 22:31:37 GMT+0800 (China Standard Time)

Themes are supposed to make use of fragment_cache (cached render result) to improve performance.
I think theme-light didn't use it.

Templates Optimization| Hexo
Fragment cache by tommy351 · Pull Request #637 · hexojs/hexo

iMil · Answer 16 · Sat Feb 27 2016 22:45:05 GMT+0800 (China Standard Time)

Perfect! here's the fix for light, in layout.ejs, replace:

<aside id="sidebar" class="alignright"><%- partial('_partial/sidebar') %></aside>

with

<aside id="sidebar" class="alignright"><%- partial('_partial/sidebar', {}, {cache: true}) %></aside>

And then:

INFO  1072 files generated in 6.44 s

Thanks a lot @leesei !

leesei · Answer 17 · Sat Feb 27 2016 22:48:40 GMT+0800 (China Standard Time)

Thanks for your testing and helping us to get to fragment_cache.
Care to submit PR to theme-light?

iMil · Answer 18 · Sat Feb 27 2016 22:49:39 GMT+0800 (China Standard Time)

Will definitely do

iMil · Answer 19 · Sun Feb 28 2016 15:19:19 GMT+0800 (China Standard Time)

@leesei PR merged hexojs/hexo-theme-light#58 (comment)
thanks for your help!

Rahil Shaikh · Answer 20 · Thu Apr 13 2017 11:49:34 GMT+0800 (China Standard Time)

Can we only generate/deploy single file?
Eg:

I run hexo deploy/generate once. (This renders everything)
Now I'm only changing the _posts (creating a new article or editing it).
Now instead of having me to do hexo generate again it would be great if I could just place that new/edited article.

mnlbox · Answer 21 · Wed May 24 2017 11:00:37 GMT+0800 (China Standard Time)

Hi guys,
I have same issue with 10000 posts. Can you make me some suggestion in related issue:
#2579

chico chen · Answer 22 · Sun Jun 11 2017 20:47:02 GMT+0800 (China Standard Time)

git clone -b source git@github.com:chilly/chilly.github.io.git
git submodules update
hexo generate

hexo generate is too slow. only 600+ posts. And cost more then 10 mins.

qixiaobo · Answer 23 · Mon Sep 18 2017 20:20:09 GMT+0800 (China Standard Time)

git clone -b source git@github.com:chilly/chilly.github.io.git
git submodules update
hexo generate

hexo generate is too slow. only 600+ posts. And cost more then 10 mins.

Me 2