npm / npm

This repository is moving to: https://github.com/npm/cli

Home Page:http://npm.community

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

stop auto-installing peerDependencies

othiym23 opened this issue ยท comments

Make all actions that schedule a peerDependency to be installed / built raise a warning.

Could we have references on this bug of the discussion that led to this? I have reservations about the idea behind this and would have preferred that only strict peerDependencies would be deprecated (let's say anything stricter than ^version is not allowed).

But this has probably been suggested & refuted at some point. I just don't know why this is not a good idea.

Thanks!

Ok, seems like most of the logic is here : #5080

I just read the first half of #5080, and I'm no more clear on what the recommended course of action is.

React Hot Loader, for instance, doesn't work if the library and the app that depends on it each have their own version of React; the library doesn't work without React, but it should be importing the same instance as your app. What's the right way to express this going forward?

So far as I can tell, the best course of action is to just leave them in there. Eventually npm will stop reading them, at which point it will cause a runtime error when require fails to find the dependency.

The current plan for npm@3 has two pieces:

  1. peerDependencies will no longer be implicitly installed if they're not already a dependency or devDependency. This is the biggest change, and is the piece most likely to require developer intervention.
  2. Invalid or missing peerDependencies will be a warning instead of an error. This will mean that irregularities with peerDependencies will be left to developers to sort out, but peerDependency incompatibilities should be less painful for consumers, because installs will bail out due to dependency problems significantly less frequently.

To be clear, peerDependencies aren't going away. The problem of keeping plugins versioned consistently with the thing to which they plug in is still a big one for many projects. The problem is that peerDependency handling often causes more problems than it solves, and has a bad habit of putting developers right back into the dependency hell that npm tries to help you escape. This is our next step to make things work more smoothly and less invasively.

Further feedback is welcome, but we're probably not going to tweak the behavior further for npm@3, which is getting very close to release.

Sounds fine to me. I don't want to be in charge of telling people what version of a library they can use - I just want to make sure I have access to the same copy they do.

Thanks for the context.

Thanks for the insight @othiym23. I've read part of the discussion and was wondering, if it will be deprecated. Now I know better.

Your arguments are very reasonable ๐Ÿ‘. I totally agree with you regarding the dependency hell and that it should be avoided whenever possible. However, it does solve particular problems very well โ€“ at least for me. And solved problems are never reported back to you guys for obvious reasons ๐Ÿ˜‰

Yeah thank you @othiym23, this is a good way to improve peerDependencies. I was scared by a full removal and you put me at ease ^^.

@othiym23 ๐Ÿ‘ Great news everyone!

@othiym23 Is there any release date for npm@3?

@th0r As is generally the case, when it's ready! It's mostly feature complete now, and @iarna is working through corner cases and bugs.

Can I clarify the changing behavior? If my existing package.json says this:

peerDependencies: {
   "foo": "*"
}

Can I update it to this and have it still work the same?

peerDependencies: {
   "foo": "*"
},
dependencies: {
   "foo": "*"
}

@getify - what do you want to "work the same"? If you want to make sure the package is included in a parent module, leave peerDependencies as is, and also specify the dependency in a parent module.

Yes, what @appsforartists said โ€“ a dependency says, "I need this thing directly available to me", and a peerDependency says, "if you want to use me, you need this thing available to you". Does that make sense?

what do you want to "work the same"?

As it stands now (before any of these changes), if someone installs my package (let's call it "bar"), they also get my package "foo" alongside it, via peerDependencies. This is important to me because my "bar" package modifies (hot patches) "foo" (with plugins), but my "bar" package doesn't provide some new module API for users to interact with -- they still need to use "foo" as patched by "bar".

If someone installs "foo" and "bar", great. But if someone accidentally installs only "bar", I want "bar" to continue to force "foo" to be side installed at the same level, so that users can still directly use "foo", even if they also require("bar") that patches "foo" with "bar" plugins.

It will still do that, but it'll do it by erroring out, rather than magically installing the package.

e.g. "bar" requires "foo", but that module cannot be found, followed by an error when require("foo") runs in bar.

Are you saying that I will no longer be able to force the "magical" side install, period? That is the only reason I use peerDependencies. I don't see how there's any point to the feature if it's not this behavior I'm discussing. It was said earlier in the thread "it's not going away". Sounds like it is.

@getify After talking to a number of maintainers of plugin ecosystems, the feedback we got was that automatically installing peerDependencies was causing the exact kinds of dependency hell problems that the feature was intended to address. It's very easy for users of Grunt, Gulp, Angular, and other tools to find themselves in a situation where their install is refusing to complete because of peerDependency conflicts and no real idea where to start to fix things. Also, it's always been a little surprising and annoying that installing, say, Yeoman globally on a system automatically globally installs Bower and Grunt as well.

The new behavior is to remove automatic installation, and to change peerDependency conflicts from an error to a warning. Yes, this means that plugin users now have to deal with ensuring that the frameworks being plugged into are installed so plugins can use them, but we feel that in the long term, developers are going to be able to figure out how to resolve version conflicts much more easily than npm could, and by making it a warning rather than an error, we introduce a certain amount of give into dealing with those conflicts.

So at this point peerDependencies is only used to throw warnings on insufficient install? I don't suppose any of the objections I would have to that are going to matter at this stage of the game, so I guess I'll just leave my packages unchanged and deal with complaints of inconvenience or confusion as they arise. Better that smaller players like me deal with such complaints than bigger players like grunt, et al.

I completely understand that this feels like a significant loss of functionality, but having dealt with a decent number of peerDependency-related issues over the last year (and having participated in the whole of the #5080 discussion), I think in the long run this will feel a lot better for many, if not most, users. It's both simpler and more explicit.

I'm very interested in getting people's feedback after they've had a chance to work with the new behavior. We'll keep tweaking peerDependencies and related features until we come up with something that's comfortable for plugin ecosystems. It just may take a couple more attempts at a solution.

I'm very interested in getting people's feedback

In the spirit of that, I filed that #7495 just now to document some ideas I've had (some of which I've actually voiced to npm team members before) about possible other ways to address the conflicts. In the future, if you're open to reconsidering deprecating and/or removing peerDependencies, perhaps that's a list of some ideas that could be discussed.

Btw: I was really surprised when I heard that peerDependencies are installed automagically. I didn't even expect this feature in the first place.

@jhnns Yes, same for me. And I strongly prefer the new version of peerDeps.

As someone who is way more familiar with peerDependency hell than I wish upon any of my enemies (we use it extensively for our DOM libraries at Spotify), I think that the solution @othiym23 wrote about above seems perfectly sensible. The well-meaning, but in practice horrible, automatic install is generally what causes almost all issues for us.

I frankly find this version of peerDependencies a lot easier to explain. Peerdependency now means - "I want this package, but I won't install it myself - I will re-use the one used by the app I'm being used in, and I'll warn if it isn't present".

I am also in favour of throwing a warning instead of erroring out. One (out of hundreds) of NodeBB's plugins used it, and we've been getting bug reports coming out of our ears ever since.

What's worse is that resolving the version mismatch still causes npm to throw errors. Maybe I'm doing it wrong, but our advice in these situations is to not fight npm, instruct the user to remove all conflicting packages, and reinstall the latest versions all in one go. I'd much rather it were a non-issue altogether.

@julianlam over 9k ๐Ÿ‘

I don't think we should fool ourselves into thinking that removing the auto-install is going to magically fix the larger problem of conflicting "dependencies" (aka version mismatch) between peers. If two packages at the same level require two different versions of the same package as a peer (not a child dep), the fact that npm didn't install one of the two versions is hardly going to do anything meaningful to fix the problem.

The warning will let you know something's wrong (but then, you'd find that problem out in other ways, if it was actually a problem), but it doesn't offer any solution.

Ultimately, removing the install may make most people feel better (at the expense of a few of us), but it's in no way a fundamental solution.


The only way to really fix this problem is not to keep wagging a finger at those of us who have needs for peer dependencies, but to actually find a way to allow co-existence of a peer in two+ versions.

One of the suggestions I made in 7495 (linked a few messages earlier) goes in that direction (using directory structure), but it'd likely require node to improve require(..) (non-breaking of course) to accommodate. I guess some feel that's impossible to expect or suggest.

But anyway, that is more like a solution. Removing auto-install is a temporary pressure-relief at best.

@getify It will be just a warning if I get it right. So it's a solution.

@zxqfox I think you and I have very different definitions of the word "solution". A warning notifies you there's a problem. It doesn't solve the problem. The thing removing auto-install does is shift the problem from one sort (conflicting installs) to another (conflicting requirements). For some people that makes them not experience the pain of the problem as directly. But that hardly qualifies as a "solution" in my book.

put another way, you're solving one smaller problem (borked conflicting installs) while just casually glancing at the bigger elephant of a problem in the corner and saying "at least I'm glad I know he's there, even if I have no solution for making him go away."

@getify, @zxqfox Could I persuade you to take your conversation to #5080? I ask this for two reasons:

  1. A lot of this is rehashing discussion that was already had there, and the participants in that discussion are in a much better place to represent the different points of view on peerDependencies than the people watching this issue, and
  2. Your debate is off-topic for this issue, which is discussing what's going to happen in npm@3, and not the ultimate disposition of peerDependencies.

Thanks!

commented

Are peerDeps actually going to be deprecated? This doesn't seem to be true judging by #6565 (comment), but a lot of people link to this issue in different repos justifying their removal by this โ€œdeprecationโ€. (This is what title says.)

Should we change peerDeps to deps in repos, or not?

@gaearon The current behavior of peerDependencies will be deprecated. See #6565 (comment) for details. peerDependencies as a section of package.json will still be around, and they'll still be a (more) useful way to indicate that a package is a plugin that needs something else to run, but there are packages that will break for sure if they continue to assume the current peerDependencies behavior.

Can we change the title to "Deprecate auto-installation of peerDependencies" for clarity?

Sorry for the late input to this discussion, but there's another use-case here: I have a module that depends on node-sass, but some users of my module want to use different versions of node-sass. With peerDependencies I can indicate the minimum version, and users will get the latest version, but if they want to use a version other than the latest then they can simply specify that in their package.json.

If I list node-sass as a dependency as well as a peerDependency will I still get the desired behaviour?

Specifically:

{ "name": "broccoli-sass-source-maps",
  "version": "1.0.0",
  "dependencies": {
    "node-sass": "^3.0.0-beta"
  },
  "peerDependencies": {
    "node-sass": "^3.0.0-beta"
  }
}

Then in a user's package.json:

{ "name": "whatever",
  "dependencies": {
    "broccoli-sass-source-maps": "^1.0.0" // uses the latest node-sass 3.0.0-beta.5
  }
}

In a different user's package.json:

{ "name": "specific",
  "dependencies": {
    "broccoli-sass-source-maps": "^1.0.0",
    "node-sass": "3.0.0-beta.4" // uses older node-sass 3.0.0-beta.4
  }
}

@appsforartists ๐Ÿ‘

@othiym23 Can we rename the issue?

"Deprecate auto-installation of peerDependencies" would be inaccurate, right? "Deprecate" != "Remove".

Suggest: "Remove auto-installation of peerDependencies"

I have a module that depends on node-sass, but some users of my module want to use different versions of node-sass. With peerDependencies I can indicate the minimum version, and users will get the latest version, but if they want to use a version other than the latest then they can simply specify that in their package.json.

@aexmachina Shortly: You should do it explicitly in any way you want to do it.

Let's go to #5080 to discuss.

@othiym23 Also I think it would be great to have a manual (few pages) with success stories of npm usage in different ways: especially for current peerDeps users.

@getify Well, agreed. "Remove auto-installation" is better.

"Stop auto-installing peerDependencies"

Simple and clear.

@othiym23 I just wanted to confirm functionality for npm@3, since I'm trying to understand the migration path for my project.

a dependency says, "I need this thing directly available to me", and a peerDependency says, "if you want to use me, you need this thing available to you".

So as I understand, in NPM3 dependency and peer dependency will function the same, with peer deps being things that should be available to users and dependencies available to the package but unavailable to user, with the only change being peer dependencies will no longer be installed?

What should be the migration plan if your project relies on several dozen peer deps? For example not just gulp but gulp plugin x y and z, all of which need to be available to the user. These will still not be available to the user if listed as a dependency still correct?

Does this mean we need to now ask users to individually install each item manually, one-by-one from the list of peer dependencies?

@jlukic As the author of the install rewrite for npm@3, let me chime in:

In all versions of npm, a dependency is guaranteed only to be available to the module requiring it.

In npm@2, a peer dependency is like injecting a dependency into the module that requires your module.

In npm@3, by contrast, a peer dependency does not result in any modules being installed. Instead, it will assert that the module requiring your module must ALSO have installed the listed peer dependency. Failure to have done so results in a warning. This is implemented here:
https://github.com/npm/npm/blob/753b6b2/lib/install/deps.js#L364-L377

It would be very unusual to ever have more than one peer dependency per module. If you have a use case for this, I'd be very interested in hearing more about it.

@iarna Emmm.

E.g.

  • Host-app with dep Plugin A
  • Plugin A with peers Plugin B & Plugin C
  • Plugin B with peer Common-v1.3-patch2
  • Plugin C with peer Common-v1.2

upd
Want to clarify a little. Let's assume that plugins is a method or functionality providers. Each plugin can provide and/or consume some methods or functionality. In that way we can have infinite tree of dependencies for some host app (to load and run them all).

I'd say Plugin A can be a module like https://github.com/c9/architect or https://github.com/zxqfox/pym or something another (but similar) service providers/consumers. This architecture can be used by a lot of existing modules (gulp-*, grunt-*, jscs-*, etc.) with some pluggable interface.

For one of my packages that uses the "plugin" model, I have a single "contrib" plugin bundle that I distribute, which has lots of plugins in it. For now I plan to keep it that way, but...

I'd very seriously considered (prior to this change) breaking that up to where each individual plugin was its own separate npm package. Many of them rely on each other, so there would be, comparatively, several peerDeps listed for each of several of those plugins.

Moreover, I would have changed the current "contrib" plugin to basically be a sort of "meta" package (or, a shortcut) which, if installed, listed all of the plugins as peerDeps. So you could have either installed individual plugins as needed, or for convenience you could have installed the "contrib" package and it would have installed all the plugins.

Bottom line: I think it's entirely possible/reasonable that some devs want their packages to have multiple (many) peerDeps. I would very likely have been going this route if npm hadn't changed to make the peerDeps unhelpful in terms of installation convenience.

@zxqfox

So let me see if I understand you right, you imagine that Host-app would directly execute the following?

var pluginA = require('plugin-a')
var pluginB = require('plugin-b')
var pluginC = require('plugin-c')
var common = require('common')

And then do things with those bits, but you don't want to have to put all of those directly in your dependencies?

(What I think is going on here is that when you say "peers" you don't mean what I mean, in so far as how the modules interact with each other. =D)

@iarna

To be more clear:

var app = new require('common');
app.use('plugin-a');
// or app.use(require('plugin-a'));
// and all the rest app (or common) will do for me:
// e.g. app.use('plugin-b'), or require('plugin-b')

app.ready(function () {
  // use any method provided by plugins somehow
});

I don't want to install them automatically because I believe we shouldn't do it with peerDependencies (or these modules). Naming don't really matters. But I want to mark packages via some simple mechanism provided by package manager (npm itself). And I feel like npm3 peerDependencies is like what I want.

Just to notify users (or warn) that packages can be incompatible.

In my case:

Common module will search for any package in host app (installed by usual dependencies) and if there are packages marked as plugins for Common (in package.json or some other file, but it's not npm issue really) it will load it by app.use or by internal dependencies.

Just usual npm deps logic is for simple modules, but not for big apps, and not for frontend apps. It's a long talk why. That's why we have bower also.

@iarna

I'll try to do a walkthrough of my specific issues which can help provide some context, as well as how I'm currently handling things.

I have a front end UI framework that requires managing permanent user files alongside the impermanent project files. These user files include files like user variables, overrides, and themes that need to be tracked by version control and cant be wiped on rm node_modules.

Installing my project with npm install triggers an interactive post-install scripts that allow user customization and triggers either an update or install based on whether the package is installed already.

The install pushes scaffolding into the root of a users project. This scaffolding includes gulp tasks which the user should then immediately be able to use for building the project themselves, and user files which should not be managed by PM. It also pushes a semantic.json file to their project root which stores a reference to their user file locations that is used to tie the node_modules install to the local install.

The peer dependencies for the project include all required user plugins to use the newly created gulpfile, and can be seen here. This makes sure that after semantic-ui is installed they can immediately run the gulp scripts and work with the project.

Since peerDependencies do not resolve before post-install script occurs in NPM@2 we also have to include some duplicates required for the install scripts in dependencies to make sure the install scripts run correctly.

You can see the original proposal for my project in this thread
Semantic-Org/Semantic-UI#1385


My concern is that removing peer dependencies without any other work around means I would have to ask the end-user to do a few dozen additional npm install foo --save commands after installing semantic-ui to work with it. This would probably stop anyone from using my library.

By the way, I wanted to say thanks for all the work you're doing, and your commitment to listening to the community. It's really appreciated. NPM has been much better than dealing with Bower or Atmosphere for handling packages, and I <3 you guys' work.

+1

I'm really glad that Peer Dependencies are being simplified rather than removed. Peer dependencies are IMO valuable for node and critically important for the web (via browserify) where singleton libraries and frameworks are the norm.

singleton libraries and frameworks are the norm.

We should not follow the norm but strive for "normal" dependencies wherever possible. jQuery plugins are basically an anti-pattern and just the symptom of a missing module system.

However, peer dependencies are sometimes necessary ๐Ÿ˜‰.

They are very useful for "global" packages that can be loaded in run-time. You know that node.js requiring all synchronously, right? ;-)

So..., we can leave grunt or gulp within peerDeps of our grunt/gulp plugins, but developer (plugin user) should install them (grunt/gulp) manually? Have I understood you aright?

I'm still really hoping for an answer about what we're suppose to do if we have many (dozen or so) peer dependencies? Ask them to install each manually?

I have never seen a project with so many peer dependencies...

They really can't be refactored to work as proper dependencies?

Or is this a case of co-requiring a dependency with something else -- they won't use it directly, but it matters that both modules use the same copy?

matters that both modules use the same copy?

I can't speak for others, but that's precisely the requirement for my usage... same instance == same copy == same canonical directory location, which thus requires it to be a peer instead of a child.

@getify - that's the same reason I use peerDependencies for my libraries. npm@3 should fix that by using flat dependencies except when there's a conflict. If authors are thoughtful in their use of semver for both publishing packages and requiring dependencies, it should be an improvement.

Yeah. In both cases, those are things that aren't exactly expressed by peerDependencies. It's a coDependency relationship, which is interesting. I personally consider it a bit of an antipattern, and bears refactoring the responsibility of those modules.

except when there's a conflict

IIUC, I think the concept of it magically (accidentally) working as a peer only until there's a version conflict then switching to a child sounds absolutely awful and just asking for confusing bugs, where it used to be sharing a singleton instance just fine and then all of a sudden it mysteriously stops working. Ugh.

where it used to be sharing a singleton instance just fine and then all of a sudden it mysteriously stops working

The risk of this happening has always existed, especially when peerDependency versions have been specified as looser semver ranges rather than pinned versions. The simple fact is that peerDependencies were intended to support plugins, and not as a means for providing singletons or shared state within JavaScript applications. There's a reason why those of us more knowledgeable about the implementation (and implications) of peerDependencies have always steered people away from using them as a way to manage singleton state (as seen repeatedly in this discussion).

I also have the same use case as @getify :

same instance == same copy == same canonical directory location, which thus requires it to be a peer instead of a child.

we have the following use case (please ask for clarification if needed):

For the purpose of centrally managing dependencies within our multi-module (componentized) web app, we have created a meta/wrapper-package that pulls in all the third-party dependencies as well as our internal core frameworks libraries, and it exports all of the dependencies for use externally. All other modules depend on this core module with somewhat loose versioning, so that they get version updates whenever the core module decides to update things. This way every new module doesn't need to add a ton of own dependencies to the package.json and version conflicts are avoided, because everything is centrally managed in the core meta-module.

Now the question is, how can we get things so that all modules share the same instance of the core module within our app, in such a way that each module can also be used independently of the app? dedupe probably helps here, but it's an additional command that needs to be run / added as a postinstall script for every module. Also we'd actually like to be warned if there are any mis-matched versions of the core module being specified -- this is something there is no real solution for at the moment.

What I think we ideally need is really something like "coDependencies" as @aredridel mentioned. Something that doesn't require a parent package, as do peerDependencies, but rather allows any number of modules to depend on one copy of a shared module, with warnings or errors in case there is a version conflict.

Of course according to the npm Roadmap "new installs are flat / deduped by default", which sounds like a great step forward. This in combination with some (tbh hack-ish) tool that checks for conflicting versions of a library in the dependency tree at run-time would probably cover our use case, albeit be significantly less elegant and easy to use than install-time version-conflict checking.

P.S. Another use case where shared dependencies are useful is in for any module that is big and involves platform-specific compilation, e.g. PhantomJS. Actually, in the case of tools like PhantomJS I can imagine it would be useful to be able to resort to the globally installed instance if available; this would make every use of e.g. karma-phantomjs-launcher 100x faster and less brittle to install...

I'm closing this as this is implemented in the multi-stage branch which will be released shortly. Further discussion should occur in new issues.