Building package dependencies with `postinstall` scripts running `npx` commands

[Michael Pontus]

We use react-jsx-parser in our project, which has package.json containing the following scripts:

    "build": "npx patch-package && yarn types && cross-env NODE_ENV=production webpack",
    "postinstall": "npx patch-package",

Unless patch-package is present in the environment when yarn tries to build the package, it will try to fetch the package from the registry and fail due sanbox restrictions:

   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR npm ERR! network
   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR npm ERR! network If you are behind a proxy, please make sure that the
   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR
   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR npm ERR! A complete log of this run can be found in:
   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR npm ERR!     /build/.npm/_logs/2022-06-09T16_51_51_997Z-debug.log
   > ➤ YN0009: │ react-jsx-parser@npm:1.28.4 [8155a] couldn't be built successfully (exit code 1, logs can be found here: /build/xfs-93d013ce/build.log)

Including patch-package in devDependencies will stop the package from trying to be downloaded, however the installed script contains a path in a shebag that does not exist in Nix environment:

   > ➤ YN0000: │ react-jsx-parser@npm:1.28.4 [8155a] STDERR sh: /build/source/node_modules/.bin/patch-package: /usr/bin/env: bad interpreter: No such file or directory

So I tried to find another way to provide patch-package into build environment and updated the derivation to include pkgs.nodePackages."patch-package" into NODE_PATH:

        pkgs.callPackage ./yarn-project.nix { } {
          src = ./.;
          overrideAttrs = old: {
            buildInputs = old.buildInputs
              ++ (with pkgs; [ python3 nodePackages."patch-package" ]);
            preConfigure = ''
              export NODE_PATH="${nodePackages."patch-package"}/lib/node_modules:$NODE_PATH"
            '';
            buildPhase = "yarn build";
            installPhase = "mv public $out";
          };
        };

Building react-jsx-parser in isolation does not aliviate the problem, as npx continues to fail to recognize local patch-package module.

        defaultPackage = pkgs.callPackage ./yarn-project.nix { } {
          src = ./.;
          overrideReactJsxParserAttrs = old: {
            buildInputs = old.buildInputs ++ (with pkgs; [ nodePackages."patch-package" ]);
            buildPhase = ''
              export NODE_PATH="${nodePackages."patch-package"}/lib/node_modules:$NODE_PATH"
              npx patch-package # ${old.buildPhase}
            '';
          };
          overrideAttrs = old: {
            buildInputs = old.buildInputs ++ (with pkgs; [ python3 ]);
            buildPhase = "yarn build";
            installPhase = "mv public $out";
          };
        };

The remaining options are patching patch-package itself or running the build in FHS user environment, and I would like to ask for a suggestion as to how this could be approached from the plugin's API standpoint.

Documentation for preConfigure hook alludes to being able to patch dependency sources before they are installed:

yarn-plugin-nixify/src/tmpl/yarn-project.nix.in

Lines 118 to 122 in 5beaf46

 # running preConfigure after the cache is populated allows for 
 # preConfigure to contain substituteInPlace for dependencies as well as the 
 # main project. This is necessary for native bindings that maybe have 
 # hardcoded values. 
 runHook preConfigure 

I would like to clarify if what should be patched are the zip files in .yarn/cache? Also, would it be a welcome addition to include similar hooks to isolated build options?

Thank you for taking time to consider my use case and for your amazing work on this project.

[Stéphan Kochen]

Hey, thanks for doing all the research into this!

I would like to clarify if what should be patched are the zip files in .yarn/cache? Also, would it be a welcome addition to include similar hooks to isolated build options?

Hmm, I'm not sure how I was using substituteInPlace when I wrote that comment. I guess maybe I was only patching the main project, not any dependency.

I have a hunch you're using nodeLinker: node-modules? I can't get npx to recognize a PnP install at all.

After some messing around, I came up with this default.nix in a small project with just those two dependencies:

{ pkgs ? import <nixpkgs> { } }:

let
  project = pkgs.callPackage ./yarn-project.nix { } { src = ./.; };
in
  project.overrideAttrs (prev: {
    npm_config_offline = 1;
    preConfigure = ''
      yarn install --immutable --immutable-cache --mode=skip-build
      patchShebangs node_modules
    '';
  })

Maybe that flow of running yarn install twice and patching node_modules / .yarn/unplugged inbetween should be standard. Better even would be if we could do it from the plugin in a single yarn install, but patchShebangs is a function loaded in the bash process, so not yet sure how we would call it from JS.

[Michael Pontus]

@stephank patchShebags did the trick, thank you!

I have a hunch you're using nodeLinker: node-modules? I can't get npx to recognize a PnP install at all.

I wasn't, originally. I only enabled it to try isolated builds. Good to know it's not an option here.