jstevans / waste-not

A caching system to help tools run incrementally w.r.t. file contents and the dependency graph

Home Page:https://www.npmjs.com/package/waste-not

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add alternative hashing mechanism for node_modules

jstevans opened this issue · comments

Currently, we don't give special treatment to node_modules -- if there's a file node_modules/myPkg/bar.js, then it gets hashed and dependency-checked the same as any other file in the repo; and if if a repo's file src/foo.js depends on node_modules/pkg/bar.js, then that dependency is presented in the graph in the same way as a dependency on another repo file.

In most projects, node_modules seems to be both much larger and slower to change than files in the repo itself. This being the case, it would be good to have a config fuzzyNodeModules that handles node_modules at the package level instead of the file level. Using the same example with src/foo.js and node_modules/myPkg/bar.js:

  • src/foo.js will be treated as depending on the node_modules/myPkg package (rather than on node_modules/myPkg/bar.js specifically)
  • there will be a single node_modules/myPkg node in the dependency graph that contains the recursive file-hash of the whole package (rather than each file in node_modules/myPkg appearing in the dependency graph)

Running this on itself (see the benchmark-processFile branch), I see:

STARTED
DONE IN 86788.95489120483ms

TOP ENTRIES:
processFile{node_modules/lodash/lodash.js} - 158.653432
processFile{node_modules/neo-async/async.js} - 101.587037
processFile{node_modules/source-map-resolve/source-map-resolve.js} - 640.555826
processFile{node_modules/ajv/dist/ajv.bundle.js} - 194.803764
processFile{node_modules/ajv/dist/ajv.min.js} - 109.212528
processFile{node_modules/ajv/lib/cache.js} - 364.88105
processFile{node_modules/ajv/lib/data.js} - 101.987961
processFile{node_modules/expect/build-es5/index.js} - 622.996264
processFile{node_modules/handlebars/dist/handlebars.amd.js} - 215.928614
processFile{node_modules/handlebars/dist/handlebars.amd.min.js} - 194.337775
processFile{node_modules/js-yaml/dist/js-yaml.js} - 116.438963
processFile{node_modules/lodash/fp/result.js} - 1527.005315
processFile{node_modules/lodash/fp/unzipWith.js} - 571.401291
processFile{node_modules/requirejs/bin/r.js} - 684.083539
processFile{node_modules/typescript/lib/lib.dom.d.ts} - 274.261523
processFile{node_modules/typescript/lib/tsc.js} - 4546.898113
processFile{node_modules/typescript/lib/tsserver.js} - 11411.919684
processFile{node_modules/typescript/lib/tsserverlibrary.js} - 11299.118504
processFile{node_modules/typescript/lib/typescript.js} - 9512.589519
processFile{node_modules/typescript/lib/typescriptServices.js} - 9707.257702
processFile{node_modules/typescript/lib/typingsInstaller.js} - 6333.728866
processFile{node_modules/uglify-js/lib/compress.js} - 123.569683
processFile{node_modules/@babel/parser/lib/index.js} - 175.728574
processFile{node_modules/@babel/types/lib/index.js} - 202.899436
processFile{node_modules/handlebars/dist/amd/precompiler.js} - 321.986059
processFile{node_modules/jsdom/lib/jsdom/browser/documentfeatures.js} - 159.730241
processFile{node_modules/jsdom/lib/jsdom/living/named-properties-window.js} - 505.641625
processFile{node_modules/jsdom/lib/jsdom/living/node-document-position.js} - 103.477281
processFile{node_modules/jsdom/lib/jsdom/living/post-message.js} - 500.039741
processFile{node_modules/jsdom/lib/jsdom/living/xhr-sync-worker.js} - 129.355988
processFile{node_modules/jsdom/lib/jsdom/living/xmlhttprequest-symbols.js} - 207.125743
processFile{node_modules/mkdirp/node_modules/minimist/example/parse.js} - 121.234851
processFile{node_modules/mkdirp/node_modules/minimist/test/dash.js} - 206.668863
processFile{node_modules/handlebars/dist/amd/handlebars/compiler/code-gen.js} - 172.938695

total of top entries: 61620.04405

total processFile{node_modules}: 83571.53660699965
total processFile{repo}: 277.72363500000006

Looks like processFile took up ~96% of time, and ~99% of that was spent in node_modules.