Some files are silently ignored when processing ~250 or more stylesheets

[Peter Magenheimer]

When running Parker on our styles directory (which is admittedly rather large), I get a discrepancy between the "Total Stylesheets" metric and the actual count of CSS files.

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ parker . | head -3
PARKER-JS
Total Stylesheets: 244
Total Stylesheet Size: 787343

peterjmag@peters-imac ~/tmp/css-metrics/flattened
$ find . -name "*.css" | wc -l
     289

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ find . -name "*.css" -exec ls -l {} \; | awk '{sum += $5} END {print sum}'
1861232

(Note that I've flattened the directory for this test case in order to rule out directory depth issues.)

In this case, it's missing 45 files. Interestingly, this number is very close to the number of CSS files in this directory that are 10KB or larger.

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ find . -size +10k -name "*.css" -exec ls -Slh {} \+ | wc -l
      43

If I remove, say, 60 files at random from the directory and run Parker again, I get accurate results:

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ parker . | head -3
PARKER-JS
Total Stylesheets: 229
Total Stylesheet Size: 1476955

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ find . -name "*.css" | wc -l
     229

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ find . -name "*.css" -exec ls -l {} \; | awk '{sum += $5} END {print sum}'
1476955

Digging a bit further, it looks like there are some file read errors that are being silently ignored. After adding some quick and dirty logging (see my patch at https://gist.github.com/peterjmag/5ff386c821a6612c99ce), I get errors like these:

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ parker .
(libuv) Failed to create kqueue (24)
{ [Error: EMFILE, open 'admintools.css'] errno: 20, code: 'EMFILE', path: 'admintools.css' }
{ [Error: EMFILE, open 'browse.css'] errno: 20, code: 'EMFILE', path: 'browse.css' }
{ [Error: EMFILE, open 'examples.css'] errno: 20, code: 'EMFILE', path: 'examples.css' }
{ [Error: EMFILE, open 'home.css'] errno: 20, code: 'EMFILE', path: 'home.css' }
{ [Error: EMFILE, open 'signup.css'] errno: 20, code: 'EMFILE', path: 'signup.css' }
[...]
PARKER-JS
Total Stylesheets: 244
Total Stylesheet Size: 787343

Perhaps Parker's just reading too many files at once? Some googling for "node EMFILE 20" led me to graceful-fs as a potential solution, so I'll try it out and submit a pull request shortly.

[Peter Magenheimer]

graceful-fs seems to fix it.

Without graceful-fs

peterjmag@peters-imac ~/tmp/css-metrics/flattened 
$ parker . | head -3
PARKER-JS
Total Stylesheets: 244
Total Stylesheet Size: 787343

With graceful-fs

peterjmag@peters-imac ~/git/parker (git::use-graceful-fs) 
$ npm link
/Users/peterjmag/.nvm/v0.10.31/bin/parker -> /Users/peterjmag/.nvm/v0.10.31/lib/node_modules/parker/parker.js
/Users/peterjmag/.nvm/v0.10.31/lib/node_modules/parker -> /Users/peterjmag/git/parker

peterjmag@peters-imac ~/git/parker (git::use-graceful-fs) 
$ parker ~/tmp/css-metrics/flattened/ | head -3
PARKER-JS
Total Stylesheets: 289
Total Stylesheet Size: 1861232