Giters

MatrixAI / TypeScript-Demo-Lib

TypeScript Library Demo Project using Nix https://matrix.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Integrate a Node native module like utp-native or leveljs in order to demonstrate vercel/pkg

CMCDragonkai opened this issue · comments

commented

The vercel/pkg has successfully been implemented and is now producing pre-release packages in our release page.

However we have yet to test how it works when dealing with native modules like utp-native.

We need to add in utp-native and leveljs as demonstrations of building a CLI executable with native code.

This will be essential for js-polykey to be capable of producing release executables.

  1. Integrate leveldb into TypeScript-Demo-Lib
  2. Integrate utp-native into TypeScript-Demo-Lib
  3. Perform a hello world equivalent of using these 2 libraries, for leveldb that's CRUD on the database, for utp-native we would need to start a server, and close the server.
  4. Then use vercel/pkg to package up the program
  5. Sanity check the release
  6. Fix up any native module configuration required for vercel/pkg

Checklist:

  • I've done some reading on native modules and vercel/pkg.
  • cloned the TypeScript-Demo-Lib, built and ran it to see it working.
  • built TypeScript-Demo-Lib with pkg and ran it in linux to check it is working.
  • Written a hello world test for level.
  • written a hello world test for utp-native.
  • pkg with level and test it works.
  • pkg with utp-native and test it works.
  • create a test for worker threads.
  • pkg with worker threads and test it works

Relevant upstream PR vercel/pkg#1273

commented

The vercel/pkg has been integrated into release.nix.

To see how to use it, see how the CI/CD uses it: https://github.com/MatrixAI/TypeScript-Demo-Lib/blob/master/.gitlab-ci.yml

commented

This has been merged upstream NixOS/nixpkgs#128967

So please also update the pkgs.nix to point to the latest master pin.

commented

Another library that is point of concern is js-encryptedfs.

The EFS uses threads.js. Which has a dependency on loading a file from its current directory.

See how in src/workers/WorkerManager it uses:

    this.pool = Pool(() => spawn(new Worker('./efsWorker')), coreCount);

This then has to load a file ./efsWorker that is relative to the script that contains that statement. Which is src/workers/efsWorker.ts.

That script will need to exist in the vercel/pkg virtual file system when it is packaged up.

We need to test js-encryptedfs inside TypeScript-Demo-Lib and see if vercel/pkg can properly package this up.

Note that there are notes about this problem in relation to webpack which has a similar problem:

We might need to do something similar for vercel/pkg.

commented

Beware of https://github.com/vercel/pkg#snapshot-filesystem and https://github.com/vercel/pkg#scripts

This might require some digging into the vercel/pkg source code.

Use it directly from npm by doing npm install --save-dev pkg. And then just use pkg by itself.

I had previously cloned the repo entirely, and was changing its source code and then calling it from that repo. That's also possible, never assume that upstream dependencies are immutable, we can always fork them and make changes to them.

commented

Progress:

  • I've done some reading on native modules and vercel/pkg.
  • cloned the TypeScript-Demo-Lib, built and ran it to see it working.
  • built TypeScript-Demo-Lib with pkg and ran it in linux to check it is working.
  • Written a hello world test for level.
  • written a hello world test for utp-native.

Still do to:

  • pkg with level and test it works.
  • pkg with utp-native and test it works.
  • create a test for worker threads.
  • pkg with worker threads and test it works
commented

I was able to get level to work. But you need to include the leveldown/prebuilds folder with the executable.
Folder structure looks like

  • linux-x64 //Executable.
  • prebuilds
    • linux-x64
commented
commented

utp-native works if I add the following lines to package.json:

{
    //...
    "pkg": {
        "assets": "node_modules/utp-native/**/*"
    },
}
commented

Can you integrate this into our CI/CD and Nix release builds? What changes would be required?
…
On 7/20/21 1:35 PM, Brian Botha wrote: I was able to get level to work. But you need to include the |leveldown/prebuilds| folder with the executable. Folder structure looks like * linux-x64 //Executable. * prebuilds o linux-x64 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE4OHN5GAIXJQLX6OFV6CLTYTVJ5ANCNFSM47Y2CL5Q.

Unless we can find a way to include this inside the executable. We will have to copy the prebuilds over a possibly create a zip archive.

commented
commented

Worker threads work fine, you just need to include the compiled worker scripts as assets in the package.json.

"pkg": {
    "assets": "dist/bin/worker.js"
  }

I have confirmed that it works on both Linux and Windows.

commented

Can you set that the worker.js file exists under src/workers in similar directory structure to js-polykey rather than in src/bin. The src/bin are meant to for executables.

commented

Anything you need my help on here? Or are we on track for merging this for this repo, and then applying it to client-refactoring branch on js-polykey?

commented

Applying the changes here to js-polykey should just be done directly on client-refactoring (assuming it all works.).

commented

Right now this is blocked by the the prebuild issue.
That is we need to have the prebuilds from leveldown bundled with the packaged executable.

vercel/pkg is meant to package them since 5.3.1 but it doesn't work for leveldown.
It does work for utp-native if we include it as an asset.
so it seems to be a bug either with vercel/pkg or how leveldown handles it's prebuilds.

A possible workaround to including the files separately is to have the packaged executable to create the needed files when it runs.
presumably we can package them as assets and have the program copy them to the needed directory at runtime.

In the meantime I've posted an issue at vercel/pkg#1264 (comment)

commented

todo:

  1. resolve the issue with leveldown.
  2. modify github-ci and release.nix depending how we solved 1.
commented

You should add a reference to #29 in the upstream issue so that people can try it out. In particular instructions on how to run it and the relevant output as well as the tree output of the build that you need to actually run.

commented

Can put this on hold until we get other issues resolved.

But you can come back to this and investigate what the exact difference between leveldown and utp-native is to see what what causes leveldown to need to exist outside the executable.

Furthermore, if it really comes down to it, we can try to do a self-extracting archive instead of a full executable. But clean up the spelling issues of the above upstream issue and we can wait for the 2nd week to start work on this.

commented

After digging through the source code of vercel/pkg I think I know what the issue is.
pkg has a dictionary directory that contains leveldown.js. theses files seem to apply patches and overrides for certain node modules. some details about this can be found here; https://github.com/vercel/pkg/wiki/Developers#dictionary

leveldown.js contains;

'use strict';

module.exports = {
  pkg: {
    patches: {
      'binding.js': ['__dirname', "require('path').dirname(process.execPath)"],
    },
    deployFiles: [['prebuilds', 'prebuilds', 'directory']],
  },
};

when the walker encounters a directory in deployFiles it doesn't include it and prints the warning we are seeing.
It seems this was not changed when the support for node files was added.

I'm looking into what changes need to be made and testing it now.

commented

Status update on progress:
At this state I am trying to test the changes however I'm running into problems with testing the vercel/pkg codebase.
building the code works fine, but testing it on Typescript-Demo-Lib it fails to package at the last stage when running prebuild-install.

The repo has been set up as such.

  1. cloned the code at version 5.3.1
  2. nix-shelled into the TypeScript-Demo-Lib
  3. navigated to the vercel/pkg code
  4. installed with npm install
  5. built code with npm run build
  6. in TypeScript-Demo-Lib I linked backed to the local pkg code with npm install --save-dev ../PathToPkg
  7. tried to build the demo with the command pkg . --targets linux-x64 --no-bytecode --public-packages "*" --output linprog --debug

With these steps we get the following output from pkg.

...
...
> [debug] prebuild-install failed[/home/brian/matrixworkspace/vercelTest/TypeScript-Demo-Lib/node_modules/leveldown/prebuilds/win32-x64/node.napi.node]:
  Error: Command failed: /home/brian/matrixworkspace/polykeyCode/pkgTesting/pkg/node_modules/.bin/prebuild-install --target v14.17.2 --platform linux --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.17.2 runtime=node arch=x64 libc= platform=linux)

And attempting to run that command with verbose to see what went wrong we get

[nix-shell:~/matrixworkspace/vercelTest/TypeScript-Demo-Lib/node_modules/leveldown]$ /home/brian/matrixworkspace/polykeyCode/pkgTesting/pkg/node_modules/.bin/prebuild-install --target v14.17.2 --platform linux --arch x64 --verbose
prebuild-install info begin Prebuild-install version 6.0.1
prebuild-install info looking for cached prebuild @ /home/brian/.npm/_prebuilds/5401a5-leveldown-v6.0.1-node-v83-linux-x64.tar.gz
prebuild-install http request GET https://github.com/Level/leveldown/releases/download/v6.0.1/leveldown-v6.0.1-node-v83-linux-x64.tar.gz
prebuild-install http 404 https://github.com/Level/leveldown/releases/download/v6.0.1/leveldown-v6.0.1-node-v83-linux-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=v14.17.2 runtime=node arch=x64 libc= platform=linux)

After some checking we find that the URL https://github.com/Level/leveldown/releases/download/v6.0.1/leveldown-v6.0.1-node-v83-linux-x64.tar.gz does not exist.

commented

Just to change track, I'm attempting to monkey patch pkg and try building the typescript demo.

commented

Confirmed, removing the override for leveldown in pkg solves the problem.
I tested this by monkey patching it in node_modules/pkg/dictionary/leveldown.js

--- a/dictionary/leveldown.js
+++ b/dictionary/leveldown.js
@@ -1,10 +1,3 @@
 'use strict';
 
-module.exports = {
-  pkg: {
-    patches: {
-      'binding.js': ['__dirname', "require('path').dirname(process.execPath)"],
-    },
-    deployFiles: [['prebuilds', 'prebuilds', 'directory']],
-  },
-};
+module.exports = {};
commented
commented

yep, tested it in WSL doing a put and a get operation. worked fine.

commented

I'm working out how to make this change when doing nix-build.
seems like I need to use node2nix to patch the code as described here https://github.com/svanderburg/node2nix#wrapping-or-patching-the-code-or-any-of-its-dependencies

I'm trying to do this but i'm still not quite up to speed with nixos. I'm looking into it.

commented
commented
commented

I've Created a PR on Github relating to this.
vercel/pkg#1273

commented

The PR for this is pretty much done.
Once it is reviewed and merged this will auto close.

commented

The npm_config_build_from_source is better than using the --build in pkg cause that would try to build node base executable from scratch and that is too complicated for now. https://github.com/nodejs/node/blob/HEAD/BUILDING.md