node-pre-gyp
node-pre-gyp makes it easy to publish and install Node.js C++ addons from binaries
node-pre-gyp
stands between npm and node-gyp and offers a cross-platform method of binary deployment.
Features
- A command line tool called
node-pre-gyp
that can install your package's c++ module from a binary. - A variety of developer targeted commands for packaging, testing, and publishing binaries.
- A Javascript module that can dynamically require your installed binary:
require('node-pre-gyp').find
For a hello world example of a module packaged with node-pre-gyp
see https://github.com/springmeyer/node-addon-example and the wiki for real world examples.
Credits
- The module is modeled after node-gyp by @Tootallnate
- Motivation for initial development came from @ErisDS and the Ghost Project.
- Development is sponsored by Mapbox
Depends
- Node.js 0.10.x or 0.8.x
Install
node-pre-gyp
is designed to be installed as a local dependency of your Node.js C++ addon and accessed like:
./node_modules/.bin/node-pre-gyp --help
But you can also install it globally:
npm install node-pre-gyp -g
Usage
Commands
View all possible commands:
node-pre-gyp --help
- clean - Removes the entire folder containing the compiled .node module
- install - Attempts to install pre-built binary for module
- reinstall - Runs "clean" and "install" at once
- build - Attempts to compile the module by dispatching to node-gyp or nw-gyp
- rebuild - Runs "clean" and "build" at once
- package - Packs binary into tarball
- testpackage - Tests that the staged package is valid
- publish - Publishes pre-built binary
- unpublish - Unpublishes pre-built binary
- info - Fetches info on published binaries
You can also chain commands:
node-pre-gyp clean build unpublish publish info
Options
Options include:
-C/--directory
: run the command in this directory--build-from-source
: build from source instead of using pre-built binary--runtime=node-webkit
: customize the runtime:node
andnode-webkit
are the valid options--fallback-to-build
: fallback to building from source if pre-built binary is not available--target=0.10.25
: Pass the target node or node-webkit version to compile against--target_arch=ia32
: Pass the target arch and override the hostarch
. Valid values are 'ia32','x64', orarm
.--target_platform=win32
: Pass the target platform and override the hostplatform
. Valid values arelinux
,darwin
,win32
,sunos
,freebsd
,openbsd
, andaix
.
Both --build-from-source
and --fallback-to-build
can be passed alone or they can provide values. You can pass --fallback-to-build=false
to override the option as declared in package.json. In addition to being able to pass --build-from-source
you can also pass --build-from-source=myapp
where myapp
is the name of your module.
For example: npm install --build-from-source=myapp
. This is useful if:
myapp
is referenced in the package.json of a larger app and thereforemyapp
is being installed as a dependent withnpm install
.- The larger app also depends on other modules installed with
node-pre-gyp
- You only want to trigger a source compile for
myapp
and the other modules.
Configuring
This is a guide to configuring your module to use node-pre-gyp.
1) Add new entries to your package.json
- Add
node-pre-gyp
tobundledDependencies
- Add
aws-sdk
as adevDependency
- Add a custom
install
script - Declare a
binary
object
This looks like:
"dependencies" : {
"node-pre-gyp": "0.5.x"
},
"bundledDependencies":["node-pre-gyp"],
"devDependencies": {
"aws-sdk": "~2.0.0-rc.15"
}
"scripts": {
"install": "node-pre-gyp install --fallback-to-build",
},
"binary": {
"module_name": "your_module",
"module_path": "./lib/binding/",
"host": "https://your_module.s3-us-west-1.amazonaws.com"
}
For a full example see node-addon-examples's package.json.
The binary
object has three required properties
module_name
The name of your native node module. This value must:
- Match the name passed to the NODE_MODULE macro
- Must be a valid C variable name (e.g. it cannot contain
-
) - Should not include the
.node
extension.
module_path
The location your native module is placed after a build. This should be an empty directory without other javascript files. This entire directory will be packaged in the binary tarball. When installing from a remote package this directory will be overwritten with the contents of the tarball.
Note: This property supports variables based on Versioning.
host
A url to the remote location where you've published tarball binaries (must be https
not http
).
It is highly recommended that you use Amazon S3. The reasons are:
- Various node-pre-gyp commands like
publish
andinfo
only work with an S3 host. - S3 is a very solid hosting platform for distributing large files, even Github recommends using it instead of github.
- We provide detail documentation for using S3 hosting with node-pre-gyp.
Why then not require S3? Because while some applications using node-pre-gyp need to distribute binaries as large as 20-30 MB, others might have very small binaries and might wish to store them in a github repo. This is not recommended, but if an author really wants to host in a non-s3 location then it should be possible.
The binary
object has two optional properties
remote_path
It is recommended that you customize this property. This is an extra path to use for publishing and finding remote tarballs. The default value for remote_path
is ""
meaning that if you do not provide it then all packages will be published at the base of the host
. It is recommended to provide a value like ./{name}/v{version}
to help organize remote packages in the case that you choose to publish multiple node addons to the same host
.
Note: This property supports variables based on Versioning.
package_name
It is not recommended to override this property unless you are also overriding the remote_path
. This is the versioned name of the remote tarball containing the binary .node
module and any supporting files you've placed inside the module_path
directory. Unless you specify package_name
in your package.json
then it defaults to {module_name}-v{version}-{node_abi}-{platform}-{arch}.tar.gz
which allows your binary to work across node versions, platforms, and architectures. If you are using remote_path
that is also versioned by ./{module_name}/v{version}
then you could remove these variables from the package_name
and just use: {node_abi}-{platform}-{arch}.tar.gz
. Then your remote tarball will be looked up at, for example, https://example.com/your-module/v0.1.0/node-v11-linux-x64.tar.gz
.
Note: This property supports variables based on Versioning.
2) Add a new target to binding.gyp
node-pre-gyp
calls out to node-gyp
to compile the module and passes variables along like module_name and module_path.
A new target must be added to binding.gyp
that moves the compiled .node
module from ./build/Release/module_name.node
into the directory specified by module_path
.
Add a target like this at the end of your targets
list:
{
"target_name": "action_after_build",
"type": "none",
"dependencies": [ "<(module_name)" ],
"copies": [
{
"files": [ "<(PRODUCT_DIR)/<(module_name).node" ],
"destination": "<(module_path)"
}
]
}
For a full example see node-addon-example's binding.gyp.
3) Dynamically require your .node
Inside the main js file that requires your addon module you are likely currently doing:
var binding = require('../build/Release/binding.node');
or:
var bindings = require('./bindings')
Change those lines to:
var binary = require('node-pre-gyp');
var path = require('path');
var binding_path = binary.find(path.resolve(path.join(__dirname,'./package.json')));
var binding = require(binding_path);
For a full example see node-addon-example's index.js
4) Build and package your app
Now build your module from source:
npm install --build-from-source
The --build-from-source
tells node-pre-gyp
to not look for a remote package and instead dispatch to node-gyp to build.
Now node-pre-gyp
should now also be installed as a local dependency so the command line tool it offers can be found at ./node_modules/.bin/node-pre-gyp
.
5) Test
Now npm test
should work just as it did before.
6) Publish the tarball
Then package your app:
./node_modules/.bin/node-pre-gyp package
Once packaged, now you can publish:
./node_modules/.bin/node-pre-gyp publish
Currently the publish
command pushes your binary to S3. This requires:
- You have installed
aws-sdk
withnpm install aws-sdk
- You have created a bucket already.
- The
host
points to an S3 http or https endpoint. - You have configured node-pre-gyp to read your S3 credentials (see S3 hosting for details).
You can also host your binaries elsewhere. To do this requires:
- You manually publish the binary created by the
package
command to anhttps
endpoint - Ensure that the
host
value points to your customhttps
endpoint.
7) Automate builds
Now you need to publish builds for all the platforms and node versions you wish to support. This is best automated.
- See Appveyor Automation for how to auto-publish builds on Windows.
- See Travis Automation for how to auto-publish builds on OS X and Linux.
8) You're done!
Now publish your module to the npm registry. Users will now be able to install your module from a binary.
What will happen is this:
npm install <your package>
will pull from the npm registry- npm will run the
install
script which will call out tonode-pre-gyp
node-pre-gyp
will fetch the binary.node
module and unpack in the right place- Assuming that all worked, you are done
If a a binary was not available for a given platform and --fallback-to-build
was used then node-gyp rebuild
will be called to try to source compile the module.
S3 Hosting
You can host wherever you choose but S3 is cheap, node-pre-gyp publish
expects it, and S3 can be integrated well with travis.ci to automate builds for OS X and Ubuntu. Here is an approach to do this:
First, get setup locally and test the workflow:
1) Create an S3 bucket
And have your key and secret key ready for writing to the bucket.
It is recommended to create a IAM user with a policy that only gives permissions to the specific bucket you plan to publish to. This can be done in the IAM console by: 1) adding a new user, 2) choosing Attach User Policy
, 3) Using the Policy Generator
, 4) selecting Amazon S3
for the service, 5) adding the actions: DeleteObject
, GetObject
, GetObjectAcl
, ListBucket
, PutObject
, PutObjectAcl
, 6) adding an ARN of arn:aws:s3:::bucket/*
(replacing bucket
with your bucket name), and finally 7) clicking Add Statement
and saving the policy. It should generate a policy like:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1394587197000",
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::node-pre-gyp-tests/*"
]
}
]
}
2) Install node-pre-gyp
Either install it globally:
npm install node-pre-gyp -g
Or put the local version on your PATH
export PATH=`pwd`/node_modules/.bin/:$PATH
3) Configure AWS credentials
There are several ways to do this.
You can use any of the methods described at http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html.
Or you can create a ~/.node_pre_gyprc
Or pass options in any way supported by RC
A ~/.node_pre_gyprc
looks like:
{
"accessKeyId": "xxx",
"secretAccessKey": "xxx"
}
Another way is to use your environment:
export node_pre_gyp_accessKeyId=xxx
export node_pre_gyp_secretAccessKey=xxx
You may also need to specify the region
if it is not explicit in the host
value you use. The bucket
can also be specified but it is optional because node-pre-gyp
will detect it from the host
value.
4) Package and publish your build
Install the aws-sdk
:
npm install aws-sdk
Then publish:
node-pre-gyp package publish
Note: if you hit an error like Hostname/IP doesn't match certificate's altnames
it may mean that you need to provide the region
option in your config.
Appveyor Automation
Appveyor can build binaries and publish the results per commit and supports:
- Windows Visual Studio 2013 and related compilers
- Both 64 bit (x64) and 32 bit (x86) build configurations
- Multiple Node.js versions
For an example of doing this see node-sqlite3's appveyor.yml.
Below is a guide to getting set up:
1) Create a free Appveyor account
Go to https://ci.appveyor.com/signup/free and sign in with your github account.
2) Create a new project
Go to https://ci.appveyor.com/projects/new and select the github repo for your module
3) Add appveyor.yml and push it
Once you have committed an appveyor.yml
(appveyor.yml reference) to your github repo and pushed it appveyor should automatically start building your project.
4) Create secure variables
Encrypt your S3 AWS keys by going to https://ci.appveyor.com/tools/encrypt and hitting the encrypt
button.
Then paste the result into your appveyor.yml
environment:
node_pre_gyp_accessKeyId:
secure: Dn9HKdLNYvDgPdQOzRq/DqZ/MPhjknRHB1o+/lVU8MA=
node_pre_gyp_secretAccessKey:
secure: W1rwNoSnOku1r+28gnoufO8UA8iWADmL1LiiwH9IOkIVhDTNGdGPJqAlLjNqwLnL
NOTE: keys are per account but not per repo (this is difference than travis where keys are per repo but not related to the account used to encrypt them).
5) Hook up publishing
Just put node-pre-gyp package publish
in your appveyor.yml
after npm install
.
6) Publish when you want
You might wish to publish binaries only on a specific commit. To do this you could borrow from the travis.ci idea of commit keywords and add special handling for commit messages with [publish binary]
:
SET CM=%APPVEYOR_REPO_COMMIT_MESSAGE%
if not "%CM%" == "%CM:[publish binary]=%" node-pre-gyp --msvs_version=2013 publish
If your commit message contains special characters (e.g. &
) this method might fail. An alternative is to use PowerShell, which gives you additional possibilites, like ignoring case by using ToLower()
:
ps: if($env:APPVEYOR_REPO_COMMIT_MESSAGE.ToLower().Contains('[publish binary]')) { node-pre-gyp --msvs_version=2013 publish }
Remember this publishing is not the same as npm publish
. We're just talking about the binary module here and not your entire npm package. To automate the publishing of your entire package to npm on travis see http://about.travis-ci.org/docs/user/deployment/npm/
Travis Automation
Travis can push to S3 after a successful build and supports both:
- Ubuntu Precise and OS X
- Multiple Node.js versions
For an example of doing this see node-add-example's .travis.yml.
Below is a guide to getting set up:
1) Install the travis gem
gem install travis
2) Create secure variables
Make sure you run this command from within the directory of your module.
Use travis-encrypt
like:
travis encrypt node_pre_gyp_accessKeyId=${node_pre_gyp_accessKeyId}
travis encrypt node_pre_gyp_secretAccessKey=${node_pre_gyp_secretAccessKey}
Then put those values in your .travis.yml
like:
env:
global:
- secure: F+sEL/v56CzHqmCSSES4pEyC9NeQlkoR0Gs/ZuZxX1ytrj8SKtp3MKqBj7zhIclSdXBz4Ev966Da5ctmcTd410p0b240MV6BVOkLUtkjZJyErMBOkeb8n8yVfSoeMx8RiIhBmIvEn+rlQq+bSFis61/JkE9rxsjkGRZi14hHr4M=
- secure: o2nkUQIiABD139XS6L8pxq3XO5gch27hvm/gOdV+dzNKc/s2KomVPWcOyXNxtJGhtecAkABzaW8KHDDi5QL1kNEFx6BxFVMLO8rjFPsMVaBG9Ks6JiDQkkmrGNcnVdxI/6EKTLHTH5WLsz8+J7caDBzvKbEfTux5EamEhxIWgrI=
More details on travis encryption at http://about.travis-ci.org/docs/user/encryption-keys/.
3) Hook up publishing
Just put node-pre-gyp package publish
in your .travis.yml
after npm install
.
If you want binaries for OS X change you have two options:
- Enable
multi-os
for your repo by emailing a request tosupport@travis-ci.com
. More details at http://docs.travis-ci.com/user/multi-os/. An example of a repo using multi-os is node-sqlite3. - Or, you can change the
language
and push to a different branch to build on OS X just when you commit to that branch. Details on this below:
OS X publishing via a branch
Tweak your .travis.yml
to use:
language: objective-c
Keep that change in a different git branch and sync that when you want binaries published.
Note: using language: objective-c
instead of language: nodejs
looses node.js specific travis sugar like a matrix for multiple node.js versions.
But you can replicate the lost behavior by replacing:
node_js:
- "0.8"
- "0.10"
With:
env:
matrix:
- export NODE_VERSION="0.8"
- export NODE_VERSION="0.10"
before_install:
- git clone https://github.com/creationix/nvm.git ./.nvm
- source ./.nvm/nvm.sh
- nvm install $NODE_VERSION
- nvm use $NODE_VERSION
4) Publish when you want
You might wish to publish binaries only on a specific commit. To do this you could borrow from the travis.ci idea of commit keywords and add special handling for commit messages with [publish binary]
:
COMMIT_MESSAGE=$(git show -s --format=%B $TRAVIS_COMMIT | tr -d '\n')
if test "${COMMIT_MESSAGE#*'[publish binary]'}" != "$COMMIT_MESSAGE"; then node-pre-gyp publish; fi;
Then you can trigger new binaries to be built like:
git commit -a -m "[publish binary]"
Or, if you don't have any changes to make simply run:
git commit --allow-empty -m "[publish binary]"
Remember this publishing is not the same as npm publish
. We're just talking about the binary module here and not your entire npm package. To automate the publishing of your entire package to npm on travis see http://about.travis-ci.org/docs/user/deployment/npm/
Versioning
The binary
properties of module_path
, remote_path
, and package_name
support variable substitution. The strings are evaluated by node-pre-gyp
depending on your system and any custom build flags you passed.
configuration
- Either 'Release' or 'Debug' depending on if--debug
is passed during the build.module_name
- thebinary.module_name
attribute frompackage.json
.version
- the semverversion
value for your module frompackage.json
.major
,minor
,patch
, andprelease
match the individual semver values for your module'sversion
node_abi
: The node C++ABI
number. This value is available in javascript asprocess.versions.modules
as of>= v0.10.4 >= v0.11.7
and in C++ as theNODE_MODULE_VERSION
define much earlier. For versions of Node before this was available we fallback to the V8 major and minor version.platform
matches node'sprocess.platform
likelinux
,darwin
, andwin32
unless the user passed the--target_platform
option to override.arch
matches node'sprocess.arch
likex64
oria32
unless the user passes the--target_arch
option to override.
The options are visible in the code at https://github.com/mapbox/node-pre-gyp/blob/612b7bca2604508d881e1187614870ba19a7f0c5/lib/util/versioning.js#L114-L127