generate_build_file.ts can fail with EMFILE

[John Firebaugh]

🐞 bug report

Affected Rule

The issue is caused by the rule: npm_install

Is this a regression?

This likely regressed in e3963e1.

Description

generate_build_file.ts performs recursive filesystem operations asynchronously without any form of rate limiting or handling of errors relating to reaching file descriptor limits.

rules_nodejs/internal/npm_install/generate_build_file.ts

Lines 689 to 705 in 0be0eeb

 const listing = await fs.readdir(p); 
  
 await Promise.all(listing.map(async f => { 
   // filter out folders such as `.bin` which can create 
   // issues on Windows since these are "hidden" by default 
   if (f.startsWith('.')) return []; 
   const pf = path.posix.join(p, f); 
    
   if (await isDirectory(pf)) { 
     if (f.startsWith('@')) { 
       await findPackagesAndPush(pkgs, pf, dependencies); 
     } else { 
       pkgs.push(await parsePackage(pf, dependencies)); 
       await findPackagesAndPush(pkgs, path.posix.join(pf, 'node_modules'), dependencies); 
     } 
   } 
 })); 

This can lead to the script failing with an EMFILE error.

🔬 Minimal Reproduction

Unfortunately this issue does not lend itself to a minimal reproduction. It's dependent on size of the node_modules folder and system-specific ulimit configuration. Also, it's non-deterministic.

🔥 Exception or Error

INFO: Repository npm instantiated at:
  /Users/mzhou/figma/figma/WORKSPACE.bazel:85:12: in 
  /private/var/tmp/_bazel_mzhou/529a40494971fd74139c7ccd5277dbfc/external/build_bazel_rules_nodejs/index.bzl:78:17: in npm_install
Repository rule npm_install defined at:
  /private/var/tmp/_bazel_mzhou/529a40494971fd74139c7ccd5277dbfc/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:786:30: in 
ERROR: An error occurred during the fetch of repository 'npm':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_mzhou/529a40494971fd74139c7ccd5277dbfc/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 784, column 24, in _npm_install_impl
		_create_build_files(repository_ctx, "npm_install", node, repository_ctx.attr.package_lock_json, repository_ctx.attr.generate_local_modules_build_files)
	File "/private/var/tmp/_bazel_mzhou/529a40494971fd74139c7ccd5277dbfc/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 490, column 13, in _create_build_files
		fail("generate_build_file.ts failed: \nSTDOUT:\n%s\nSTDERR:\n%s" % (result.stdout, result.stderr))
Error in fail: generate_build_file.ts failed: 
STDOUT:
STDERR:
node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^
[Error: EMFILE: too many open files, open '_/node_modules/@types/babel__generator/package.json'] {
  errno: -24,
  code: 'EMFILE',
  syscall: 'open',
  path: '_/node_modules/@types/babel__generator/package.json'
}

🌍 Your Environment

Operating System:

Output of bazel version:

Rules_nodejs version:

5.5.2