run `prepublish` for git url packages
isaacs opened this issue · comments
After installing a git url package, perhaps the prepublish script should be run, since git repos will typically not keep around any built artifacts that the prepublish script generates.
However, prepublish scripts also typically require the use of devDependencies packages, so it might not be possible to do this in a good way. (In which case, it'll just be on the users of git url packages to run whatever scripts are necessary to make them ready to use.)
I don't believe it's necessary. If the user has forked a project and is installing via a raw git repo, they could add a commit to cause preinstall to run the prepublish script if needed, and leave it alone if they don't.
I am with @dfellis after thinking 2 days about this and I am still not sure haha.
My thoughts are:
- if a package is promoted to be installed via git, this would be unnecessary anyway. the developer will make sure it works.
- if a package is actually in the registry but installed via git, the user knows what he is doing and will do as @dfellis mentioned (or any other solution)
- the
devDependencies-problem sucks,prepublishis meant to be run on the developer-side
But I think it should be mentioned in the docs maybe?
use case:
- condition: package needs to be compiled before installation, e.g. from coffeescript to javascript
- condition: compiled files are not welcome in the repository
- goal: allow developer to publish package in registry in a compiled state, so common users do not need any pre/postinstall
- goal: allow users to install a different versions or forks of the package from git repos without having to do anything special (e.g. as a project dependencies)
sample current solution by @paulmillr at https://github.com/brunch/brunch:
- on prepublish compile all and manually turn off postinstall in package,json
- on postpublish reenable postinstall
- on postinstall (which runs only when installed via git, as it is disabled in packages in registry): compile all
limitations:
- clearly devDependencies will need to be installed for packages with prepublish installed via git repo ... this is ok as long as users did really need to run preinstall, in this case it is the right thing to do
- complex prepublish scripts may depend on other development tools, that may not be available on user machines
options:
- don't do anything, let developers solve that like in https://github.com/brunch/brunch ... imho awkward
- run
prepublishon git install ... will fail packages with complex prepublish scripts installed via git repo (but currently they were just installed unbuilt anyway) and force people to simple, or all-js scripts (may be both a good practice, and maybe too restrictive in some cases) - add and run
gitinstalllon installing via git repo, installgitDependenciesin that case ... adds new settings, but breaks no compatibility and solves the issue, my 2c
I agree. npm install with no args already runs the prepublish script for the same reason, so it would make sense to do the same for git urls.
I also agree. Repositories are for the source files, why should we have to add extra bulk (especially for larger packages)?
Since the idea of version control is to control the source, npm should assume that only the source is contained in a git repo and run prepublish when installing a package from git. It is theoretically identical to installing a package in a local directory.
Any updates?
npm has 861 open issues and is a bit behind at handling them. It will take a while to respond.
I personally would appreciate if I didn't get notifications about people inquiring about updates, but, alas.
Yes, between this and #1876 I can't figure out any good way to use a forked module in dependent project, without publishing it to the public repo.
Backstory: I'm trying to get https://github.com/natevw/node-hid/tree/dhleong/Node-0.11.13 building for atom-shell on Windows. Somethings odd, and I'm trying to simplify how the build process works, but between these two issues there's not really a good way depend on git submodules in a non-published node module.
@natevw: try npm shrinkwrap, which is infuriating to use in the longer term, but can help with what you're describing.
@braco Thanks for documenting that, it is similar to what I ended up doing (which was to make a GitHub release tarball and install that instead).
By the time a Git URL's source code makes it to my node_modules, the repo's .npmignore has been applied.
the developer will make sure it works.
I don't get the full source from which to build. So I can't make sure it works. Not within package.json, anyway.
Any word on this?
I ran into this issue yesterday and… ended up just making a coffeescript-less compiled branch: fregante/jade2php@4cf629a
npm i --save-dev 'bfred-it/jade2php#precompiled'
Shouldn't installing from Git URL work exactly as cloning a repository to a temporary directory, installing all dependencies and then installing from there? Installation from local directory already triggers the prepublish, which is very handy.
What would be the downsides to this approach?
@lzrski +1
Imo, running prepublish on install from git repo is best solution. It is second most compatible solution. First one would be adding special gitinstall script, but it's probably to much effort for simple case and would need all projects to be updated, so I do not consider it viable path. They're both discussed at #3055 (comment).
The only drawback is it may fail with too complex prepublish scripts with non-js dependencies, but IMO it doesn't matter, as 1. they are rare, and 2. this update won't break anything atm, either they publish compiled js files already and disable prepublish in the published package (=> will continue to work), or they were just installed unbuilt from git repo (=> they didn't work already)
Thanks @iki. I din't mention it, but OFC you proposed this approach before.
I would add another use case:
You develop an app (possibly proprietary), and want to modularise it and put some reusable code to their own (maybe open source) packages. You develop these packages along your main app and when deploying it to your staging server you want the app to install latest revision of your packages (e.g. from master branch). But you do not consider this code production ready (you want to test it on stage before) and don't want to put it to npmjs yet. So you use Git URLs in your app's package.json. With prepublish triggered on such installation your deployment scripts could be much simpler.
We just run into this today, while trying to migrate a soon-to-be-open-sourced repository from a local directory to a private github repo. When updating other project dependencies on this, we were very surprised to find out that while declaring the dependency as a local filepath runs prepublish, declaring it as a git url does not. Our surprise for this was related to #3059, these two behaviors seem at odds with one another.
My proposal for the next major version change to npm is to add new lifecycle hooks "sourceinstall" and "localinstall" to differentiate from "install" which is most commonly used when installing a module from npm.
- If a dependency is referenced as an npm module or tarball url, continue as per usual.
- Otherwise, treat the git url, github url or local file path dependency as a source dependency. To avoid over-loading "prepublish", I would propose a "presourceinstall" to allow for different behavior in this case.
- Do not run "prepublish" after
npm installwith no args, replacing with "postlocalinstall".
+1 "sourceinstall" sounds nice, I get this problem a lot when installing private packages from git which require a prepublish hook like babel transpiling.
Hello, @leebyron and @madjam002! Thanks for your participation.
I'm not sure if there is a need to introduce new hooks. Could you provide some use cases for this new syntax that wouldn't be covered by just running prepublish on git install?
I would argue that this solution doesn't add complexity and is backward compatible as @iki mentioned.
The big thing for me at least, is that we at nodegit have code that needs to be run BEFORE compilation, but after dependencies are installed (we're actually generating the C++ for our module). Right now, we just have a script that runs an npm install --ignore-scripts and then does it's stuff, and then calls out to node-pre-gyp to see if a precompiled binary is available, and then finally compiles if needed.
@lzrski One thing I've noticed which could be a flaw with both the prepublish hook and the proposed sourceinstall hook is that if you have an .npmignore which excludes your untranspiled source code to prevent it being packaged up, doing an npm install with a git repo means that the source directory will be excluded. If your prepublish hook ran something like babel to transpile the source code, it wouldn't be there when it goes to run it on install!
To be fair, this would be solved by removing the source from npmignore, but a lot of existing packages which require a build step with something like babel do have the source code npm ignored, and it would be nice to find a solution which works with most existing repos. (Let's say I forked a package and made some changes and wanted to npm install it using git).
Would it be bad to suggest that installing through git ignores the contents of .npmignore and then it can just run the prepublish hook? Or we could maybe clone the git repo into a "staging" area, run prepublish against that, and then copy the contents across excluding the files from .npmignore into node_modules. We could then discard the contents of the staging area.
@madjam002 The solution I've settled on for that issue is that rather than using an npmignore, is to have a script for prepublishing (another reason prepublish is potentially not the best solution here) that gets everything ready, generated/transpiled etc, then tarballs just the files I want included. No npmignore to cause the issue, that way, but you still get to filter what's included in the standard install.
@maxkorp Sounds cool, but how would this work if I were installing a module that did this via git? With the increasing use of build steps I think it's important to get consistent behaviour between installing from wherever, whether that's the file system, git or the npm registry, and it would be good to have that behaviour inside of npm.
@madjam002 re your yesterdays comment:
The last option you gave (with staging area) is what I was thinking about. It could work by executing git clone <git-url> <temp-dir> and then npm install <temp-dir>.
The prepublish step is already executed when installing from local directory. I'm not familiar with NPM's internals, but I guess it wouldn't require much coding. We would just reuse existing logic. I'd be happy to roll out a PR with it, but I'm not sure where to start. Any advise?
@lzrski Yeah that's exactly what I had in mind, should be fairly simple! I have no idea about the internals of npm either so I can't be of any help at the moment even though I'd like to help with this in some way. I'll see if I can get a better understanding of what's going on under the hood next week some time.
@madjam002 thanks for clarification. Let's stay in touch.
@maxkorp I'm trying to grasp your reservation toward this approach.
AFAIK when installing from local dir, first all the dependecies are installed (including dev dependencies) and then prepublish script is executed. So whatever you do to your code, you could put it in prepublish script and it will have the dependencies ready.
Maybe your prepublish logic requires external tools that are not available via NPM? Then you can just ask your team or contributors to install required software prior to installing your package via git. Please consider that installing stuff from git repos is not meant to be standard distribution method. It is only meant to be used in development or staging environments. In production you will still probably use npmjs.org or private NPM repository with everything prepared.
BTW nodegit looks really cool. I was looking for something like that.
Just to start, here is my understanding of the order of things when installing inside the local dir, in case I have something wrong.
If you don't have install scripts:
- Install dependencies
- Compile
- Run prepublish script
If you do have install scripts:
- Run preinstall script
- Install dependencies
- Run install script
- Run postinstall script
- Run prepublish script
- Run preinstall script
Because i need the code that would be run as prepublish to run BEFORE install, because the code generated by that script is the actual code being compiled. I'm not generating javascript from coffee,
I'm generating the actual C++ native addon code from templates.
What I need is this order:
- Install dependencies
- Run some post-dependency-install-pre-compilation script (to do what you're proposing I use prepublish for, if I understand correctly)
- Run install script
- Run postinstall script
Currently, during install I run npm install --ignore-scripts to get all my dependencies, then run the same script called by prepublish. My issue with that is that --ignore-scripts also causes submodulesto not run and compile either, if I understand correctly. I haven't bumped into that yet, but I've also been deliberately avoid the situation. The alternative is to read the package.json in a script, and manually callnpm install {packagename}` for each package.
After we've gotten ready to compile, we shell out to a build tool (pangyp, specifically) to do the compilation. A separate but related issue would be that if we could have a preinstall script do something like the generation i need, while allowing NPM to continue compilation, rather than having to manually take over.
I realize that this is an oddball situation and most people are not generating any of (much less the majority of) their C++ code, so it might not be an issue everybody wants to worry about, since I can work around it, and I can totally respect that. However, It seems like an issue that keeps coming up for other people as well in various forms, so if the work is going to be done anyways, I figure I ought to at least push to get our scenario handled as well.
My use cases: testing against a #develop branch, or installing the equivalent of a dev snapshot from #master.
For that, if a git-hosted install has a prepublish script, then ALL dependencies (including devDependencies) should be installed and then prepublish should be run. It should be like if someone git cloned the repository and ran npm install inside it.
Some projects can be installed directly from git without running a build or fetching devDependencies. I can't think of a reason reason such projects would even have a prepublish script, so these steps wouldn't need to be followed if no prepublish script exists.
@rconnamacher there's very good reasons such a project would have a prepublish script: Re-run tests to abort if it fails to keep bad/flappy code out of NPM, build documentation for the newly anointed version of the library, confirm the CHANGELOG file has the specified version listed as a section of the document, etc.
Anything you'd want to happen ONLY before you publish to NPM.
Thanks @maxkorp for sharing your scenario. It is complicated indeed. I probably need some time to get a grasp on it.
In the mean time I've put a dummy package to investigate the behavior of npm install executed in various ways. There are also logs. Please play with it.
From my observation the only significant difference in installing from Git vs from local directory is that prepublish is not being executed with Git. So it doesn't seem like solving this issue is a big change and it would greatly help with common cases like compiling sources or the one @rconnamacher mentioned.
Maybe we should keep it's scope as narrow as possible? I.e. move the discussion about introducing new hooks to new issue?
@dfellis I wouldn't like to derail the conversation, but I think what you describe fits better into preversion hook. The prepublish is used by most people for anything you want to happen before the code is prepared for deployment. It does not necessary involve publishing the code, so maybe the name is a little misleading. Please note that npm won't let you publish script with the same version number, so you are forced to bump version before running npm publish. On the other hand you don't want to force developers to update changelog and docs whenever they want to install their code in dev, staging or CI environment.
The name is very misleading, and it's basically being used incorrectly by most people if that's the case.
I've argued this position literally for years now, but I don't expect sanity to ever come to NPM on this subject, which is a shame because otherwise it has such a powerful and comprehensible design.
How about this: clone the repo to a temporary directory, run npm install && npm pack there (which means pulling all dev dependencies), then untar the package tarball we just generated to the node_modules directory in question, and fetch (or perhaps move) any non-dev dependencies it has.
- The repository is treated as a source repository, which means you'll have to assume that all dev dependencies and all prepublish scripts might be needed. For every point and purpose, building from that temporary directory is just like what you'd do prior to publishing, which should work if the package is sufficiently portable.
- The permanently installed part is like a published package, though, with everything those have, but without anything
.npmignoregot rid of, and without unneccessary dev dependencies. For every point and purpose, the installed package behaves just like what you'd get from the registry if it were installed there.
@gagern A nice clean solution, I think that would work well.
I actually like that very much. Great idea!
Inspired by @gagern I've published a simple script that you can run in install hook that roughly emulates what he proposed. Please take a look: lzrski/npm-git-install. It is meant as temporary solution for this issue.
By default it relies on non-standard gitDependencies section of package.json, but user can avoid this with command line arguments. There is also an API for build systems.
Comments are welcome, but please use issue tracker of my project. We don't want to add noise to this discussion :)
I finally took the time to find my way around the codebase, and wrote an implementation of what I suggested earlier. See #11092 for the pull request. In the process I found and fixed some other git-related issues, namely #11091 fixing an often-duplicated issue and #11087 which I haven't seen a report for yet. Let's hope that the next npm release improves the situation for git dependencies quite a bit.
@denvned i'd like to point out to you that your commit message is displayed on this thread and is not considered constructive or appropriate. i imagine there is something more useful you could use, for both your own team and the community at large. in the future, please avoid this kind of attitude and language. thanks.
@ashleygwilliams Sorry, I didn't mean it. NPM is a great tool. I just wish someday NPM will treat Git dependencies as they usually are, i.e. as source repositories, instead of treating them as prebuilt package hostings.
Currently, for example, to depend on a not yet published version of a third party package that needs to be compiled, one cannot just specify Git URL of a needed branch in the repository, but have to fork the repository, build the package, commit built artifacts to the fork, and use Git URL of the fork instead of the original repository. And if there is any update to the original branch, one have to repeat the whole procedure. It is not strikingly convenient...
P.S.
I'm glad to know someone from the NPM team actually monitors this thread. 😉
+1 for a fix here. Between this and babel/babel-loader#149 I'm having a really difficult time making changes to third-party dependencies that are needed for my project.
As far as I can tell, #11092 (comment) indicates what currently prevents #11092 or #7040 from fixing this. NPM mixes command line arguments and config settings into one global data structure, so if you did npm install -g foo then any git dependency of foo would be installed globally as well. There could be other undesirable config setting propagations as well. Fixing those is what requires more work, likely a lot more.
Please add support for prepublish on git urls.
Hi, a few questions:
- Is there currently a clean workaround to this issue?
- What are the tasks one can contribute to help fixing this issue?
@Volune please consider my workaround: lzrski/npm-git-install. It's up to you to decide if it's clean.
@Volune @lzrski also see this simple gist script I wrote. It's more hacky than @lzrski's solution but might help you out, it's working for me at the time being. https://gist.github.com/sankho/e11894de0618684a8159b39804abe565
any updates ?
Ditto @Volune. Or rather, a better way to phrase it: why does this still have the patch-welcome tag? Because, frankly, a patch fixing this (for the reasons mentioned by @gagern / #11092 (comment), covered by #11092 / #7040) would involve re-writing half of npm; so it seems like this can't be completed by someone external to npm?
@lzrski your solution worked great, just I have severe issues getting it to work in docker containers.
@blocka glad to hear that it works for you. Can you open an issue in my project and elaborate about your problems with docker?
@lzrski your solution looks great, but it doesn't seem to work in my case: I'm using it for coffeescript compilation, and have dependent project setup to run coffeescript compilation at the prepublish stage:
"prepublish": "coffee -c --output lib src"The depending project is configured to run npm-git-install at the install phase, which I believe is what you're supposed to do.
I do see the compilation in the temporary folder kicking in, and I did actually check that the output is there, (the lib folder does exist and it does contain the compiled .js files), but somehow they are not getting included in the corresponding node_modules folder.
Am I missing out on something here?
Hi, I think Git should only be considered as source repositories, instead of treating them as prebuilt package hostings.
But GitHub has an extra feature called :
GitHub Releases
About the GitHub releases :
Releases are GitHub's way of packaging and providing software to your users. You can think of it as a replacement to using downloads to provide software.
In a GitHub release you can add a custom binary : A tar.gz that can be generated with npm pack during continuous integration.
I tried to do that with GitHub but a faced an NPM install error when downloading the tar.gz
❯ npm install https://github.com/user/repo/releases/download/v1.0.0/bin.tar.gz
npm ERR! fetch failed https://github.com/user/repo/releases/download/v1.0.0/bin.tar.gz
npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 404
PS: The URL github.com/user/repo/releases/download/v1.0.0/bin.tar.gz is redirecting with a 302 status and a HTTP location header set to https://github-cloud.s3.amazonaws.com/releases/XXX/XXX...
I posted a SO question about that error: http://stackoverflow.com/q/38242147/1480391
Should I create an NPM issue about that?
@wspringer can you open an issue at https://github.com/lzrski/npm-git-install/issues? Let's try not to add to the noise here. If you do, please let me know what version of npm you are using and maybe paste some logs.
Thank you :)
This might be relevant: https://github.com/exogen/postinstall-build
Ticket almost 4 years old and not resolved yet 😢
@lzrski your solution works but it is painfully slow, but thanks anyway
@aleclarson Does that package work as a workout for this bug?
+1 for this
When doing a lib fix/PR, it is currently quite painful to use your own fork in your own project. In my case I had to publish my fork under a new package name so that it's easy to integrate in other client apps.
As transpilers like Babel are becoming quite popular for most libs the current situation is painful
Related #10074
I would expect following: when dependency installed from git and there is prepublish hook, install dev dependencies for this repo and run hook. If there is no prepublish hook, no need to install dev dependencies, just git clone.
This is now available in npm@5.0.0. instead of prepublish, you'll need to use prepare, but this is otherwise as you'd expect. 🎉
@zkat Thanks a lot, this is really great! Any more detailed info on how to run prepare? Maybe documentation? Thank you.
@involve-it I guess this could be better documented. But essentially, prepare will run if you install a git dependency that has a prepare script. You don't need to do anything but add a prepare script to its package.json.
@zkat I am excited to have the new prepare feature, but it crashes NPM for me. I'm looking forward to it working for me!
Same here @trusktr. Seems like npm does not install devDependencies before running the prepare script.
I'm running into this problem too-- it appears that for me, running npm@5, if I install a package A that has a git dependency B, and that package B has a prepare script, it's not getting run at all. If I instead use a postinstall script, it appears to run but it doesn't have the devDependencies, so it fails. If I try to use a prepublish script, it doesn't appear to be run at all either. Pretty frustrating, I just want to work on a bundled library and a consumer of that library at the same time without publishing them.
prepare will run if you install a git dependency that has a prepare script. You don't need to do anything but add a prepare script to its package.json.
This doesn't appear to be the case for me:
"scripts": {
"dev": "npm run babel -- --watch & npm run gulp -- watch & npm test -- --watchAll & wait",
"build": "npm run babel && npm run gulp",
"babel": "babel src --out-dir lib --ignore spec.js,test.js",
"gulp": "gulp css",
"test": "jest --coverage",
"lint": "eslint src",
"format": "prettier --single-quote --trailing-comma=es5 --write src/**/*.js",
"precommit": "npm run format && npm run lint && npm run test && git add src",
"prepare": "npm run precommit && npm run build"
},"dependencies": {
"my-package": "git://ghe.spotify.net/user/my-package"
}npm i my-package installs the package but does not build. No lib directory at all, only src.
I'm running into this problem too-- it appears that for me, running npm@5, if I install a package A that has a git dependency B, and that package B has a prepare script, it's not getting run at all. If I instead use a postinstall script, it appears to run but it doesn't have the devDependencies, so it fails. If I try to use a prepublish script, it doesn't appear to be run at all either. Pretty frustrating, I just want to work on a bundled library and a consumer of that library at the same time without publishing them.
@mturley, how did you manage to solve this problem?
If a project is going to be able to build an npm package from source, wouldn't it always need to install devDependencies (some) and peerDependencies for that package. Doesn't look like npm does that currently.
If NPM isn't, then it definitely should.
prepare will run if you install a git dependency that has a prepare script. [...]
This doesn't appear to be the case for me:
Same here. prepare only runs before packing and publishing, and when you npm install inside the package itself. It does not run for dependencies, only preinstall, install, & postinstall do.
Here is a package you can use to test this and see what scripts get run: https://github.com/AlexanderOMara/test-package-json-scripts
I don't think the problem described in this issue is actually solved, although it was closed.
@AlexanderOMara We're facing the same issue here.
To use, prepare runs with npm install only on Mac OS 10.12.6, but not on Red Hat Linux or Ubuntu/Debian. Any thoughts?
EDIT: we're on npm5 + node8
@kafkahw AFAIK, prepare script are not supposed to run for dependencies at all, on any platform (I tested on macOS 10.12.something). I don't know if/how you got a different result.
@AlexanderOMara I created a test repo that demonstrates npm install might trigger dependency's prepare script. Feel free to check it out here: https://github.com/kafkahw/npm-install-test.
Basically, npm-install-test has a dependency called node-package1 that has a prepare script calling a missing method. When I npm install package npm-install-test, I got an error about that missing method from prepare script of dependency node-package1.
I tested this on Mac OS 10.12.6
@kafkahw Well, now I'm even more confused... I too get the prepare script failed issue using your repo. Beyond that though, I can't make sense of what is going on.
If I npm install on this package:
{
"name": "test-module-git",
"dependencies": {
"test-package-json-scripts": "git+https://github.com/AlexanderOMara/test-package-json-scripts.git"
}
}Then node_modules/test-package-json-scripts/log.txt looks like this:
2017-11-14T00:29:50.592Z: preinstall
2017-11-14T00:29:50.698Z: install
2017-11-14T00:29:50.798Z: postinstall
Leaving no sign that prepare ran on the dependency (I'm not sure if it actually ran or not).
If I use yarn install instead though, I get a node_modules/test-package-json-scripts/log.txt that shows prepare was run:
2017-11-14T00:19:00.952Z: preinstall
2017-11-14T00:19:01.076Z: install
2017-11-14T00:19:01.185Z: postinstall
2017-11-14T00:19:01.292Z: prepare
2017-11-14T00:27:09.047Z: preinstall
2017-11-14T00:27:09.156Z: install
2017-11-14T00:27:09.258Z: postinstall
Tested using macOS 10.12.6, npm 5.4.2, yarn 1.3.2.
At this point, I'm really confused as to how this is suppose to work and why Yarn seems to work in my case, but NPM does not.
@AlexanderOMara I logged an issue #19152 to track this problem.
prepare seems to be working for me now, packages from git are built properly, at least the ones I just tried. Maybe it's been fixed in newer version of npm?
P.S., first time ever installing a package on a machine takes a while; I'm guessing it installs dev dependencies because that's the only way it can possibly build the package. The second time installing the package is fast, from the cache in ~/.npm.
@erikfox I'm facing the same issue. Need to transpile code after fetching it from a private git repo.
Build script works correctly on the repo directly, but not when it gets installed into node_modules.
Only one file in the dist not the whole src directory file structure.
Were you able to resolve this issue? Not sure if it's an npm's problem or babel's.
Allright. Got it to work. "files" in package.json and a non-obvious npm behavior regarding .npmignore and defaulting to .gitignore were the key for me.
Basically if you don't have .npmignore it uses .gitignore in its place and usually you don't want to commit your dist and have it in .gitignore...
Adding "dist" to files in package overwrites all ignores and does not cleanup after the build.
Here are the docs for all of the future adventurers.
FWIW prepare doesn't appear to execute when installing a git dependency from a branch on npm@latest
@shellscape file a new bug and provide detailed environment details/steps to reproduce