Add a require-bin or require-tools section to reduce the need for composer global

Question

Add a require-bin or require-tools section to reduce the need for composer global

greg-1-anderson opened this issue 8 years ago · comments

With an empty composer.json:, when I run these commands:

composer global require "laravel/installer=~1.1"
composer global require wp-cli/wp-cli

I get this output:

Using version ^0.23.1 for wp-cli/wp-cli
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for wp-cli/wp-cli ^0.23.1 -> satisfiable by wp-cli/wp-cli[v0.23.1].
    - Conclusion: remove symfony/config v2.2.11
    - Conclusion: don't install symfony/config v2.2.11

(Error output truncated)

And I expected this to happen:

Installing commandline tools following recommended installation procedures on the specified project's installation instructions page should succeed, and not be subject to variable failures depending on which other projects were installed first.

Analysis

The Composer documentation says:

Composer is not a package manager in the same sense as Yum or Apt are. Yes, it deals with "packages" or libraries, but it manages them on a per-project basis, installing them in a directory (e.g. vendor) inside your project. By default it does not install anything globally. Thus, it is a dependency manager. It does however support a "global" project for convenience via the global command.

However, the design of Composer is clearly geared towards the per-project dependency management model. The global keyword creates a "global" project as a convenience; however, the users of this keyword do not expect that the projects they install in this way should all become part of a single project. Each project installed has its own set of requirements, and its own expectation that these should be managed in a self-consistent way, and yet, if certain projects are unceremoniously combined, then unexpected project conflicts may be encountered. These errors are difficult for beginning Composer users to understand, and they return with them to the maintainers of the project they are trying to install.

Suggested Solutions

In the short term, stop promoting the use of composer global require as an installation method for global cli tools, and instead, actively discourage its use. As an alternative, users should use composer require to install each commandline tool to its own local project, and manage their $PATH or binaries manually (e.g. by creating symlinks from a bin directory already in the $PATH).

In the mid-to-short term, the Composer global keyword should be deprecated (e.g. add warning messages when it is executed), and then either removed or fixed.

If the global keyword is retained, it should not continue to be used to manage a "global" project, as doing this is untenable. Instead, it should be re-cast as (or replaced by) a mechanism for managing a global bin directory. This is the primary use-case for it right now; it is feasible and practical to combine bin directories from disparate projects without causing conflict, as long as each project had its own vendor directory -- as is the fundamental assumption for top-level Composer projects.

For example, a "global" but isolated project could be installed to ~/.composer/global/[something]; its vendor and bin directories would appear in their usual locations, and the contents of the ~/.composer/global/[something]/bin directory could be mirrored (via symlink) in ~/.composer/vendor/bin or, perhaps a better option would be simply ~/.composer/bin. There are various ways that the string [something] could be chosen; the most straightforward would be simply org/project (although this means that long paths such as ~/.composer/global/org/project/vendor/org/project would exist).

One complication of the proposed solution is that further management of the "global" projects would be more complicated. With the global keyword, it is possible to run composer global require foo/foo, followed later by composer global require foo/extension, and the extension will end up in the same project as the top-level foo/foo project. This is an uncommon use-case, though; usually, the global keyword's only use is for the installation of atomic global commandline tools.

I could work on a PR to provide global bin directory management, if there is some agreement on how these commands should best work.

Christophe Coevoet · Answer 1 · Wed Jun 01 2016 16:33:05 GMT+0800 (China Standard Time)

This perfectly reflects the reasons why I never use composer global require

Johannes Haseitl · Answer 2 · Wed Jun 01 2016 18:23:31 GMT+0800 (China Standard Time)

@greg-1-anderson, for that issue, you might also need to have a look at how composer handles global plugins and installers. I tried to explain the problems with the current implementation in #5333.

Rob · Answer 3 · Wed Jun 01 2016 21:31:48 GMT+0800 (China Standard Time)

I am in favor of deprecating global. It is unlikely that this will happen though.

As for proposed solutions, I see no need for any. Most projects that can be installed globally, are also available in Phar form. The latter can easily be "installed" somewhere in your PATH.

Christophe Coevoet · Answer 4 · Wed Jun 01 2016 21:37:21 GMT+0800 (China Standard Time)

And for your suggested solution of managing several separate projects and linking their bin scripts to a common folder in the path, I suggest you to build this as a separate tool (using the composer CLI under the hood to install dependencies) instead of having it in composer itself.

Not that changing the behavior of composer global to be something else cannot happen (at least until 2.0) for BC reasons

Jeff Geerling · Answer 5 · Thu Jun 02 2016 02:36:08 GMT+0800 (China Standard Time)

There are still valid use cases for 'global' packages... but since composer global require is basically a minefield of problems in real-world usage, I would tend to agree that, unless it ever becomes less of a minefield, it would be nice to officially deprecate its usage.

With my Ansible roles, probably 1/3 or more of the issues I get for composer, Node/NPM, and Ruby/gems all have to do with people screwing things up with regard to global vs project installs/dependencies.

Pedro Cordeiro · Answer 6 · Thu Jun 02 2016 03:01:44 GMT+0800 (China Standard Time)

Instead of removing global installs altogether, it would be nice if composer showed a relevant warning message and asked for confirmation before installing a project globally. Something like:

Installing libraries globally is not recommended, as it may create conflicting dependencies with other  installed libraries. Install anyway? [y|N]

So people who know how/when to use it won't miss it, and people will (hopefully) stop advising for its use.

Bruno Škvorc · Answer 7 · Thu Jun 02 2016 04:03:38 GMT+0800 (China Standard Time)

I'd like to chime in on the other side of the spectrum and say that global require saves me a bunch of time very often. If you're using best practices and keeping your main development machine clean, only developing in VMs or containers, you'll never run into this problem, but will allow for lightweight project folders (and better IDE performance to boot), shared testing tools with setups common to your workflow, and more.

In short, I see nothing wrong with global because the development environment of every app I'm working on is isolated. I would argue that developing everything on one machine (virtual or not) should be discouraged sooner than discouraging the use of global require.

I do agree with @naroga though in that there should be a warning saying "installing globally might create conflicting dependencies", but without the "is not recommended" part.

Marco Pivetta · Answer 8 · Thu Jun 02 2016 06:59:37 GMT+0800 (China Standard Time)

If you're using best practices and keeping your main development machine clean, only developing in VMs or containers, you'll never run into this problem

So it is not a global installation ;-)

Running a docker container with a modified $PATH (or just the entry point) and a local installation would be equivalent and still better than something as painfully against the composer advantages as composer global

Taylor Otwell · Answer 9 · Thu Jun 02 2016 11:42:19 GMT+0800 (China Standard Time)

Totally agree. It would be amazing to have each composer global installed package installed into its own isolated directory with its own isolated dependencies instead of possibly conflicting with other globally installed packages.

Greg Anderson · Answer 10 · Sat Jun 04 2016 23:19:48 GMT+0800 (China Standard Time)

There is a use-case where composer global require does in fact serve a purpose vis a vis global dependencies, and that is the use of global Composer plugins. This tends to be uncommon; one example I found of this is francoispluchino/composer-asset-plugin, which recommends installation via composer global require in order to extend Composer to allow for the management of non-php assets. This plugin could also be added on a per-project basis, where it is needed. Whether or not global composer plugins should or should not be supported indefinitely in Composer is an open issue. I tend to think that since Composer is designed to manage dependencies for a single project, that global plugins should also be deprecated and ultimately removed. For the time being, though, I would like to focus on replacing the use of composer global require for commandline tools that have no need whatsoever to be combined.

Predominantly, the projects found in the top Google search results that recommend the use of composer global require do so for the purpose of installing exactly this sort of commandline tool. This practice can, in fact, be actual harmful, not like "putting orange juice in your gas tank" harmful, as one commenter put it, but all the same causing appreciable lost time and frustration in a number of circumstances. #5333 is a prime example, where a commandline tool is installed globally per one of its recommended installation methods, and it in turn brings in composer installer into the Composer global scope, when said installer was only intended to apply to specific local projects. If this is an XY problem, then it seems that the question of "what are you trying to do?" is "install a global tool safely", and the solution to that problem is "do not use composer global require" to install commandline tools.

There are only a few instances where composer global require is actually safe. If a project has NO dependencies, then it can always be safely installed globally. This is exceedingly uncommon; most of the commandline tools surveyed are based on symfony/console, which brings in a number of other additional dependencies as well. A big question really is, why isn't composer global require already an even larger problem than it currently is? The reason that composer global require is predominantly working is that there is overlap between the users of these different tools, and the tool maintainers are going to extra effort to ensure that their projects are functional when installed in a global context with other tools. Take the Larevel installer as one example of this; it maintains compatibility with three different versions of Guzzle. An installer has no need to be combined with other projects; efforts to make this possible are a waste of resources that would not be necessary if the installer was properly isolated. Composer is designed to provide exactly this sort of isolation, and the isolation is broken only due to the global command, which is used in this case only to make it easier to install the tool.

For those who like the convenience of composer global require, and would also like to be able to maintain the per-project isolation that Composer usually provides, I have thrown together a prototype tool called cgr (because it replaces composer global require). It would be interesting to hear feedback from folks who use composer global require to find out if this alternative also meets the needs of their use cases.

Greg Anderson · Answer 11 · Tue Jun 07 2016 00:26:35 GMT+0800 (China Standard Time)

Since folks love composer global require, perhaps we should continue to support it. The one difficulty we have is that, for some use-cases (global composer plugins -- rare), compser global needs to operate on a common "global" project, and for most other use-cases (installing a commandline tool such that it's binary scripts are available on the user's $PATH -- most common scenario), the "global" projects should be installed in separate projects, as Composer assumes they should be.

New Proposed Solution

composer global require --isolated behaves like the cgr script.
composer global require --shared behaves like the current composer global implementation.
--isolated prints out a warning message that Composer plugins will not be globally available in this mode, and --shared should be used if this is intended.
--shared prints out a warning that the dependencies of all globally-shared projects will be combined, and --isolated should be used if this is not intended (or if problems are encountered).
Composer 1.x defaults to --shared if no flag is specified; Composer 2.x defaults to --isolated.

These flags could be named in other ways, of course, e.g. a single flag with a value indicating "shared" vs. "isolated". Whatever flag name or names is/are used, they should be chosen to avoid conflicts with flags used in other Composer commands that may be called via composer global.

Bruno Škvorc · Answer 12 · Tue Jun 07 2016 00:32:40 GMT+0800 (China Standard Time)

I tested CGR and was quite happy with how it worked. I tried to encourage others to do the same and provide feedback (should yield a user or two I hope), but I think it's a good solution, if global require is really causing problems for some people. I wouldn't mind having cgr's functionality replace the default functionality in version 2.0 of Composer, and I wouldn't mind at all having to use cgr until then.

However, the --isolated and --shared scenario sounds rather good, too. Excellent ideas all around @greg-1-anderson

Renat Golubchyk · Answer 13 · Tue Jun 07 2016 01:24:23 GMT+0800 (China Standard Time)

Why does this problem exist in Composer and not in similar tools like NPM. What do they do differently here?

Marco Pivetta · Answer 14 · Tue Jun 07 2016 01:26:08 GMT+0800 (China Standard Time)

Why does this problem exist in Composer and not in similar tools like NPM. What do they do differently here?

Dependencies in composer are all in the same set (flat structure, basically), while in NPM, since JS has no globally defined types, each package requires all its dependencies as sub-dependencies, recursively.

Bilal Amarni · Answer 15 · Tue Jun 07 2016 01:32:35 GMT+0800 (China Standard Time)

There might be use-cases between isolated and shared. For example, one could want to require some behat contexts alongside behat/behat so they can be autoloaded automatically when running behat.

Also, these concerns about dependency conflicts with cli tools are also valid for local projects (for the record I had also started a plugin for this : https://github.com/bamarni/composer-bin-plugin).

Yassine Guedidi · Answer 16 · Wed Jun 08 2016 05:45:42 GMT+0800 (China Standard Time)

I'd suggest:

composer require x/y to require as a dependency of the current project (current behavior)
composer global require x/y to require a global dependency, for composer plugins (current behavior)
composer global bin-tool x/y as a special require command that isolate package dependencies

Greg Anderson · Answer 17 · Sat Jun 11 2016 02:20:18 GMT+0800 (China Standard Time)

If we added require-bin as a compliment to require-dev, then composer require --bin or composer global require --bin could add the specified projects to the require-bin section, whereupon their dependencies could be installed in an isolated location. This would satisfy @bamarni's desire to isolate binaries in local projects, in addition to global projects, and @yguedidi's desire to maintain the current behavior of composer global require.

--no-dev could apply to require-dev and require-bin projects, but it would probably be better to add an independent --no-bin flag.

Bilal Amarni · Answer 18 · Sat Jun 11 2016 15:58:30 GMT+0800 (China Standard Time)

Some concerns if we go into this direction :

there might be non-dev cli tools (eg. doctrine/migrations), so we'd need require-bin and require-dev-bin
only describing cli dependencies as an isolated single packages would be limited. With Behat for example it is very common to register the Mink extension, a Mink driver, some custom contexts.
we'd have to go through all commands / behaviours and adapt them for the new composer.json schema.

I think this would require some massive changes as composer.json has always been about describing a single set of dependencies, and here we'd have multiple. So changing the json schema for this sounds very unrealistic imo. It has also been rejected not long ago (#4381).

Greg Anderson · Answer 19 · Tue Jun 14 2016 02:12:35 GMT+0800 (China Standard Time)

Yes, makes sense; we should try to move forward without composer.json changes.

Sam · Answer 20 · Tue Jul 26 2016 16:09:14 GMT+0800 (China Standard Time)

A lot of discussion focuses on resolution of incompatible dependencies; should we also support aliasing to support different versions of a binary tool?
For example:

composer require-bin phpunit/phpunit "4.8" as phpunit48

Rasmus Schultz · Answer 21 · Thu Aug 11 2016 19:48:48 GMT+0800 (China Standard Time)

composer global operates on a user-specific global composer.json file and (as mentioned above) is necessary and meaningful for things like global Composer plug-ins - even if that's not a very common thing.

Conceptually as well as practically, this is a very different from creating isolated projects for the purposes of installing packages and their dependencies in isolation. I'd argue this feature has nothing to do with composer global in the first place, since it has nothing to do with the user-specific global composer.json file.

What the cgr script does is something very distinct from this - it's actually managing creation, updates and removal of isolated projects.

I would therefore like to propose an entirely different set of commands, distinct from global - for example:

composer get phpunit/phpunit:4.8        # create project and install version 4.8
composer get phpunit/phpunit            # create project and/or update to latest version
composer drop phpunit/phpunit           # uninstall and delete the project

The get command would create a dedicated project for the package, if not already present, write the appropriate constraint into the require section of the composer.json of the dedicated project, and then composer update.

Running the get command with just the package name would basically just composer update the dedicated project for the package, without changing composer.json first; if the dedicated project doesn't exist yet, it would default to a version constraint of * and then composer update.

The composer drop command would composer remove the package from the dedicated project (to make sure bin files are uninstalled) and then delete the dedicated project.

Thoughts?

Carlos Jimenez · Answer 22 · Thu Aug 11 2016 20:04:32 GMT+0800 (China Standard Time)

I think is a bit confusing see: "require" and "get" maybe is better use "require isolated project/project" or "require general project/rpls"
My vote goes to better naming rather than new commands

Rasmus Schultz · Answer 23 · Thu Aug 11 2016 20:50:12 GMT+0800 (China Standard Time)

Overloading the require command, which operates on your local repository, IMO is just as bad as overloading the global command, which operates on a user-specific repository - this new feature operates on neither of those, it actually creates repositories. Overloading either of those is therefore, in my opinion, going to be much more confusing and surprising than simply adding new commands, which are then more obviously offering completely separate functionality.

Greg Anderson · Answer 24 · Thu Aug 11 2016 22:17:47 GMT+0800 (China Standard Time)

I agree with @mindplay-dk here. While I am uncertain what the best and clearest names for these commands should be, that is just a semantic issue; I do think that the "install application" operation needs to be logically separated (i.e., implemented as a separate command) from require, as he suggests above.

Rasmus Schultz · Answer 25 · Fri Aug 12 2016 18:36:23 GMT+0800 (China Standard Time)

Thinking more about this though, it occurs to me there might be cases where you want to do something different to one of your isolated projects. For example, if you've installed some tool like, say, phpunit or phpmd, it's not unthinkable you might want to install an optional plugin of some sort as well, is it?

So maybe this actually needs to work more like global, allowing you to issue any command against a specific isolated project with a specified name?

Greg Anderson · Answer 26 · Sat Aug 13 2016 01:08:12 GMT+0800 (China Standard Time)

Yes, that is true -- an application that you install globally might want to have plugins as well. I have been considering an interim command in cgr to extend an installed application by running composer require in its installation directory. The thing is, though, that this operation requires a lot of parameters, because you need to identify the project you want to extend, and the project(s) you want to add to it. @Saphyel's suggestion is still a possibility, but you'd still need to decide how extending a project works. Does require isolated a/a b/b install two projects, both isolated, or does it install a/a as an isolated project, and extend it with the project b/b? To answer @mindplay-dk's question about any command, maybe we could have:

composer isolated a/a install
composer isolated a/a require b/b
composer isolated a/a remove b/b
composer isolated a/a remove

In other words, an install or require or remove without any sub-projects listed could install or remove the main application itself.

In the meantime, folks can always use the alternate solution shown on the cgr page -- i.e., manage your project directory manually, and use cd to select the project to extend.

Ed Reel · Answer 27 · Thu Oct 20 2016 09:05:43 GMT+0800 (China Standard Time)

First off, I'd like to thank @greg-1-anderson for his excellent work on the cgr script and many other contributions to open source in general. Maybe I'm oversimplifying the underlying issue here a bit but what is wrong with just removing the global keyword altogether and having every project install in an isolated directory under ~/.composer and have every binary installed in a common ~/.composer/bin directory? It seems impractical to keep adding new paths, symlinks or aliases to binaries for different "isolated" projects. This would be the most pragmatic solution in my mind.

Rasmus Schultz · Answer 28 · Thu Oct 20 2016 15:03:55 GMT+0800 (China Standard Time)

@uberhacker that breaks the case where I'm installing a command-line shell project and several packages implementing actual commands. Just an example - point is, you can't assume somebody doesn't want multiple packages in a single project, anymore than you can assume that they want all packages in a single project.

How about this: change the default behavior so that composer global commands are always running in a specific "global project" scope - but when you don't specify the project scope, it defaults to "default".

So for example:

composer global require foo/bar installs into the project folder named "default".
composer global:default require foo/bar does exactly the same thing.
composer global:hello require foo/bar installs into a project folder named "hello"

When a project folder doesn't exist, a composer.json is auto-generated, and then bin option is pointed to e.g. ../bin, so that all your binaries end up a single folder you can add to your system path, same as today.

In other words, the only real change here, is the global project path is one level deeper than it is today, and defaults to "default" - things should continue to work more or less precisely the way they work today, but with the ability to have several global projects on demand.

We'd still need a few additional commands, e.g. global-list to show a list of existing global projects, global-remove to uninstall a global project, etc. (these aren't strictly needed, but would be pretty handy, as opposed to digging around in the folder yourself.)

This way, the meaning of "global", in a Composer context, remains the same as it is today - the only difference being there's a "default" global project, and you can create others.

Thoughts?

Ed Reel · Answer 29 · Thu Oct 20 2016 19:29:31 GMT+0800 (China Standard Time)

@mindplay-dk: I think maybe you are misunderstanding what I'm suggesting here. When someone types composer require org/project, I'm thinking rather than installing the project in the current directory, install it in ~/.composer instead and just put any binaries from the project in ~/.composer/bin without using the global keyword at all. I think it might make sense to add the shared keyword and install all these projects in a single directory such as ~/.composer/shared but in a way that other projects can access. But I'm in favor of removing the concept of global and how it works currently altogether.

Rasmus Schultz · Answer 30 · Fri Oct 21 2016 00:07:11 GMT+0800 (China Standard Time)

Reviewing OP:

The global keyword creates a "global" project as a convenience; however, the users of this keyword do not expect that the projects they install in this way should all become part of a single project.

I don't think that's necessarily true. I've always understood global to be a single project into which packages may be globally installed. The realization that this would created conflicts came with time of course, but yeah.

Each project installed has its own set of requirements, and its own expectation that these should be managed in a self-consistent way, ...

Not necessarily - there are cases where multiple different packages plug into a command-line shell project as add-ons. If we make the scope of "global" packages unique per installed package, that is no longer possible.

...and yet, if certain projects are unceremoniously combined, then unexpected project conflicts may be encountered. These errors are difficult for beginning Composer users to understand, and they return with them to the maintainers of the project they are trying to install.

I think this is the biggest problem.

If one does not wish to use global installs, one doesn't have to - deprecating "global" completely doesn't change the fact that you could install all your tools locally.

In some cases, a local installation doesn't make sense though. A good example of that is Composer itself, or any other truly stand-alone command-line utilities; there isn't necessarily anything to gain by installing a copy of those for every project.

There is also a potential ambiguity issue, e.g. projects that install command-line utilities that you might need, but aren't themselves dependencies of the project you're installing - in this case, the require section contains potentially misleading information about the true requirements.

There is are also potentially situations where you want to update a command-line utility and it's dependencies, but not your project - in this case, having a global installation you can update without affecting your project's "composer.lock", and/or avoiding potential conflicts with your project's dependencies, can be useful. (for example, if your tool needs and update to a/b^2, but your own project requires a/b^1 and you aren't ready to upgrade.)

I think the least disruptive thing would be to make a conceptually backwards-compatible change, e.g. allow for multiple global projects with separate dependencies.

Greg Anderson · Answer 31 · Fri Oct 21 2016 00:24:20 GMT+0800 (China Standard Time)

@mindplay-dk That was sort of what I had in mind, except my thought was to use composer global org/project <command> to identify the global namespace via the primary project that goes in it. global:<id> seems okay to me too, but it might be inconvenient to parse that with Symfony console.

Rasmus Schultz · Answer 32 · Fri Oct 21 2016 03:56:49 GMT+0800 (China Standard Time)

@greg-1-anderson the reason I proposed it that way, is that org/project unnecessarily forces you to pick a vendor and package name every time - I'm just one user, so there's no risk of collision between vendors or packages; one level is enough, e.g. could create local projects alays with a vendor-name of global, e.g. global/{project}, requiring you to specify only a project-name.

it might be inconvenient to parse that with Symfony console.

Looks like the find() method is how Application resolves the command-name - override the method and see if the command name starts with global:, if so, e.g. $this->commands["global"]->setProjectName($name) to inject the project-name, then return $this->commands["global"] and let the command dispatch as normal. (if the command doesn't start with global:, just return parent::find($name) and let it do what it normally does.

Mike Pirog · Answer 33 · Sat Oct 22 2016 06:54:30 GMT+0800 (China Standard Time)

👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍

Ed Reel · Answer 34 · Sat Oct 22 2016 07:44:22 GMT+0800 (China Standard Time)

Maybe I should clarify what I envision as my ideal composer require(install?) org/project behavior. I'm thinking any projects should install under ~/.composer (not the current directory like it works today) without the global keyword (let's just nix that concept altogether) with the structure as shown below:

.
├── bin
│   ├── project1 binary
│   ├── project2 binary
│   └── project3 binary
├── org1
│   ├── project1
│   ├── project2
│   └── project3
├── org2
│   ├── project1
│   └── shared
│       └── project2
├── org3
│   └── project1
└── shared
    └── org1
        └── project1
    └── org2
        └── project2

So, for example, if I wanted to install an "isolated" project, I would enter:
$ composer require org1/project1
If I wanted to install a shared project within an organization, I would enter:
$ composer require org1/shared/project1
If I wanted to install a shared project across all organizations, I would enter:
$ composer require shared/org1/project1
Basically, it would create the directories just as you enter them in the command. Of course, composer would need to be re-engineered so it could share projects in this way but it makes sense from an architectural standpoint. Thoughts?

Einar Gangsø · Answer 35 · Sat Oct 22 2016 08:19:44 GMT+0800 (China Standard Time)

@uberhacker how does this handle versions?

Ed Reel · Answer 36 · Sat Oct 22 2016 12:53:27 GMT+0800 (China Standard Time)

@Elgan: I'm not sure it would be any different than it works currently?

Greg Anderson · Answer 37 · Sat Oct 22 2016 22:34:23 GMT+0800 (China Standard Time)

I think that we should restrict discussion in this issue to the behavior of composer global. Changing the existing cwd behavior of non-global composer is a radical change; that proposal should be brought up in a new issue.

Mike Pirog · Answer 38 · Tue Oct 25 2016 02:23:59 GMT+0800 (China Standard Time)

Mike Pirog commented 8 years ago

Rasmus Schultz · Answer 39 · Tue Oct 25 2016 04:05:31 GMT+0800 (China Standard Time)

I think that we should restrict discussion in this issue to the behavior of composer global.

Agreed, 100%.

$ composer require shared/org1/project1

I find this to be ambiguous and very confusing - when I see the slash, I expect that to be a composer vendor/package ID. When I see two slashes in the same string, I get confused. It took me a minute to even understand what was being proposed. This syntax is odd.

Also, other commands, like update and remove etc. are important - that's why global works nicely for what it does at the moment, managing a single repository: it's not really a command, it's a pseudo-command that can dispatch any other command by simple changing to a different working directory.

I'm strongly in favor of enhancing the existing global command.

Concretely, I'd propose:

Fully preserve existing behavior for backwards compatibility: the default for global should be the same working directory as today, otherwise upgrades will break your existing working global installs.
With something like global:phpunit, it should then switch from the default (same as today) working directory to a different one.
In either case, it should auto-initialize your composer.json project files, e.g. adding the appropriate bin-dir setting.
Running composer global without arguments should display a list of projects.
Removal of projects? (or just expect users will delete created folders by hand? hmm.)

In terms of documentation, I would propose to refer to these as "global projects", e.g. the current standard global project is the "default global project", and additional user-defined projects are "named global projects", in order to minimally detour from the existing familiar terminology.

Greg Anderson · Answer 40 · Tue Oct 25 2016 04:35:51 GMT+0800 (China Standard Time)

otherwise upgrades will break your existing working global installs.

I'm not sure that I agree that there is such a thing as an existing "working" global install. A global install that happens to be working today may become a non-working install at any moment that any of the currently-installed projects are updated. composer global update becomes a perpetually dangerous command.

I think that in the future (Composer 2.x), backwards compatibility should be broken. The existing composer global command should be reserved for composer plugins (e.g. composer-versions-check) that must exist in the Composer "global" project. Perhaps composer should reject (i.e. immediately fail with an error) if composer global require is used on a project that does not contain a Composer installer.

I'm not entirely sold on the syntax of global:phpunit at the moment, but this proposal is entirely the right direction: there should be some way to "globally" install multiple command line tools such that they have separate dependencies, but share the global bin directory (`~/.composer/vendor/bin).

In the short term, composer global require should continue to be used to install global Composer plugins, and cgr should be used for command line tools.

Bruno Škvorc · Answer 41 · Tue Oct 25 2016 05:04:33 GMT+0800 (China Standard Time)

I'm not sure that I agree that there is such a thing as an existing "working" global install. A global install that happens to be working today may become a non-working install at any moment that any of the currently-installed projects are updated.

I disagree with this thought. To me, this is like saying "all forks are dangerous because if you put them into the ground upright and run through that field you're going to get hurt eventually". Well yeah, but use the fork for food like you're supposed to. Use global in environments that can be inherently isolated in regards to global packages, i.e. the aforementioned vagrant setup, and you're good.

Greg Anderson · Answer 42 · Tue Oct 25 2016 05:22:44 GMT+0800 (China Standard Time)

Metaphors are not always analogous to the situation the purport to model. In the case of composer global require, requiring multiple unrelated projects will work if (a) their dependencies are non-overlapping, or (b) their dependencies are mutually compatible. In the case of (a), barring any projects adding new dependencies, composer global update can continue to work indefinitely without problem. In the case of (b), on the other hand, it is likely that at some point one of the overlapping dependencies will come out with a new major release. When that happens, it is even more likely that one of your global projects using that dependency will upgrade before all of the others (that is, as opposed to all of your global projects adopting the new major version at exactly the same time). That is the "dangerous" situation that could cause composer global update to stop working at any moment.

Since novice users can't readily distinguish between (a) and (b), saying "don't run with scissors" is not helpful. As for your aforementioned vagrant setup, it shouldn't break your usage model if you had to switch from composer global require phpunit/phpunit to something equivalent to composer global:phpunit require phpunit/phpunit just once, when upgrading to Composer 2.x, if such a change were adopted, whereas that same change would save a lot of time and trouble for a lot of users.

Lynna Landstreet · Answer 43 · Thu Jul 06 2017 03:25:26 GMT+0800 (China Standard Time)

I found this post while trying to troubleshoot some problems I was having installing a couple of command-line tools (PHP CodeSniffer and PHPCompatibility) via Composer. But if I'm understanding this correctly, Composer should not be used for things like that (i.e. tools that you want to be able to use on a variety of projects), only for specific projects?

I'm a little confused by this - the whole reason I even have Composer installed is that it was required in order to install another tool recently, Mixed Content Scan, which like the others above is not project-specific. Both of the ones I've been trying to install today give Composer installation instructions, and don't seem like the sort of thing you would want to install separately for every project you work on - I mean, at a glance, I have 40 local Git repositories for various projects, and I certainly don't want to have to install every useful tool 40 times over!

Should I not be using Composer to install any sort of commands that I want to be able to use globally? Is doing so invariably going to cause some kind of problems?

Greg Anderson · Answer 44 · Thu Jul 06 2017 03:37:56 GMT+0800 (China Standard Time)

@lynna-freeform Yes, your summary is correct. A lot of global php commandline tools recommend using composer global require because it is easy. Unfortunately, using composer global require for more than one such tool can cause problems. The way forward in this issue is unclear; until a solution is provided in Composer itself, global commandline tools should recommend using cgr to install.

Bilal Amarni · Answer 45 · Thu Jul 06 2017 04:26:10 GMT+0800 (China Standard Time)

For the record there is also my plugin : https://github.com/bamarni/composer-bin-plugin

It has a different approach than cgr. While it might be less opinionated / user-friendly, it also solves more use cases :

installing cli tools locally without mixing them with project's dependencies
adding dependencies to cli tools. This is useful for exemple with behat or any other cli tool which supports plugins / extensions. With these tools we can usually define custom autoload rules through their config file, but it's more convenient to have them autoloadable out of the box.

Greg Anderson · Answer 46 · Thu Jul 06 2017 04:45:41 GMT+0800 (China Standard Time)

bamarni/composer-bin-plugin is very nice; I added a link to it from the cgr README.

I see development on bamarni/composer-bin-plugin started a few weeks before this issue was posted. If I had known about it, I might not have bothered to write cgr.

David C. Campbell · Answer 47 · Tue Sep 19 2017 10:45:15 GMT+0800 (China Standard Time)

There seems to be quite the variety of solutions building up around this:
Phar.io, CGR, ComposerBin, Tooly, Scoper, etc...

With the explosion of PHP CLI tools in the past couple years (primarily as PHARs), is the simplest solution not for Composer to simply have something like a binary dist type, where-by the URL is simply downloaded and moved/linked into the bin-dir w/o extraction? I know there's been similar suggestions before, but I have to say it seems like the most straight-forward solution that alleviates if not completely solves the core issue in most real-world use cases.

Rasmus Schultz · Answer 48 · Wed Sep 12 2018 17:09:04 GMT+0800 (China Standard Time)

Having to pick a third-party tool for this really is a nuisance.

I don't know why there's anything controversial about adding this feature.

I'd like to propose a really simple solution: just add an option to define the home folder, and make it accept a relative path.

So:

composer global require foo/bar by default installs to the COMPOSER_HOME folder as today.
composer global require foo/bar --home=blaz installs to {COMPOSER_HOME}/blaz
composer global require foo/bar --home=/usr/bin/blaz installs to /user/bin/blaz

As far as I can tell, the only thing you would need to do is join the COMPOSER_HOME path with the -home argument, if present (using normal path resolution rules) and if the folder doesn't exist, emit a composer.json with bin-dir set to ../vendor/bin so that binaries end up in the usual folder.

This approach is simple, backwards-compatible, has practically zero learning curve, should be very easy to implement, and provides complete freedom to install any combination of tools and plug-ins etc. as needed.

I know there's a gazillion tools to address this need already, but I don't even want to choose, much less have to direct users to one or more choices of tools so they can install my tools - and clearly the existence of these tools demonstrates a need.

Please, can we get past the FUD and just solve this?

Sam · Answer 49 · Wed Sep 12 2018 17:19:58 GMT+0800 (China Standard Time)

@mindplay-dk you can already do it inline if you want:

COMPOSER_HOME=blaz composer global require ...

Rasmus Schultz · Answer 50 · Wed Sep 12 2018 20:16:20 GMT+0800 (China Standard Time)

@SamMousa that doesn't "just work" - you'd have to modify your search path every time.

if the folder doesn't exist, emit a composer.json with bin-dir set to ../vendor/bin so that binaries end up in the usual folder

Nils Adermann · Answer 51 · Thu Feb 14 2019 21:24:47 GMT+0800 (China Standard Time)

Looking through this issue and the various comments on here I think we're actually conflating two key problems, so I'll try and summarize the two main points as I see them and add my opinion where i have one.

1 - What is the purpose of requiring a package "globally" rather than locally in a project?

For Composer plugins it's the only way to modify the behaviour of commands like composer init or composer install before project specific plugins are installed. This is not necessary for custom installer plugins which simply change the target directory for code, but is sometimes necessary for some of the more involved plugins changing Composer's behaviour altogether.
- I'm personally not a big fan of plugins making major modifications to how Composer behaves, but there are a couple so there does appear to be a need for this, I'd be all for deprecating/disallowing these, but I think we'd face a lot of resistance
For PHP tools/binaries the goal is make a single installation available across projects.
- Personally I think this is entirely unnecessary. The only thing you're saving with shared tools is a little bit of disk space and we've already made the decision that it's more valuable to separate projects clearly and potentially duplicate installed libraries if you manage multiple projects

2 - How can PHP Tools be allowed to internally use libraries in versions which conflict with my project code or other tools I am using?

Currently global installation is used as a workaround. Global installation works as an additional project which has its dependencies resolved independently of the current project. I imagine this is why a lot of tools recommend global installation. This however has a number of issues:
- If you install multiple tools they can conflict with each other
- If you need different versions of the same tools for different projects you cannot install them because there is only one global project
Some PHP tools, like phpunit, load your project code at runtime into the same scope as their internal libraries. If your project uses a different version of a library, which the tool (e.g. phpunit) also uses, there will be a conflict at runtime. This problem will not go away by installing phpunits dependencies in a different directory than your project's dependencies. This is not a problem for tools which do not directly load your code, e.g. a static analysis tool which only inspects your code, but doesn't load and execute any of it, won't suffer from this problem.

Analyzing these problems

So to me it seems like the global command flag could continue to work as-is for global plugins, but we need to provide better solutions for installing PHP tools locally in a project. Global plugins all have to run together in the same process scope as Composer, so they should have all their dependencies analyzed together. But for tools we could strictly discourage the use of global and display warnings when using the command, if we provide better solutions suitable for tools installed on a project level.

One potential (partial) solution would indeed be to introduce a new type of require for individual tools, as has been proposed in some comments here. Packages listed in "require-tool" (not sure about the naming) would each be resolved as if they had their own composer.json requiring just them. If the tools have a plugin mechanism and need users to add more packages, we would need to provide a mechanism to group packages to resolve their dependencies together.

This solution would not yet resolve the problems with tools which load and execute project code. The only way to avoid conflicts with these would actually be to modify names when writing the tool to disk. So I think our only hope to resolve this is this project: https://github.com/humbug/php-scoper However due to PHP's dynamic nature this does not easily work on every project. So we would need a way to mark packages as compatible for use with this tool in composer.json.

Using PHAR files does not by itself fix anything, because the classnames would still conflict at runtime just the same. PHAR files are a workaround for our current lack of a "require-tool" option, but it doesn't fix runtime conflicts. We could enforce the installation of "require-tool" as phar files, but I'm not sure if there's really a benefit to doing that?

Conclusions

We should introduce a new "require-bin" or "require-tool" key in composer.json (potentially a matching -dev pair as well, if we do not want to make assumptions regarding tools and the --no-dev flag). This key should contain groups of packages with version constraints to be resolved independently of the project, so long as all dependencies are marked compatible with humbug/php-scoper. Each tool should be installed into a separate folder vendor/tools/...

We should then discourage the use of composer global require for PHP tools and display appropriate warnings.

Note, if you paid attention, you'll have noticed this particular proposal for keeping "global" around would still allow you to install tools globally into ~/.composer/vendor/tools/ and potentially even multiple versions at the same time depending on how exactly we define "require-tool"/"require-bin".

PHPUnit progress on using php-scoper: sebastianbergmann/phpunit#2015 - sebastianbergmann/phpunit#3086

Théo FIDRY · Answer 52 · Mon Feb 18 2019 02:17:40 GMT+0800 (China Standard Time)

1 - What is the purpose of requiring a package "globally" rather than locally in a project?

For Composer plugins it's the only way to modify the behaviour of commands like composer init or composer install before project specific plugins are installed.

It has been my usage thus far. The best example is hirak/prestissimo.

For PHP tools/binaries the goal is make a single installation available across projects.

Personally I think this is entirely unnecessary. The only thing you're saving with shared tools is a little bit of disk space

I don't necessarily think the need here is to save a bit of disk space but rather but some commands/tools globally usable. For example:

$ composer global require psy/psysh
# Provided you exported the global Composer bins to your PATH:
$ psysh

That said I agree the current implementation/usage is limited.

Before extending on the potential solutions, I would like add another angle to the problem.

The root of the issue is that Composer is being used as a tool installer in some cases, because it's convenient and we don't have anything better. For example you cannot have brew install deptrac or apt-get install deptrac, even if deptrac is meant to be used as a standalone tool. Indeed even though the application deptrac could be downloadable as a PHAR, you will have the issue of:

Making sure the PHAR can be executed: it's just a compressed PHP file so it still needs PHP. And so far there is no way to export a PHAR bundled with a static PHP binary (and even in this case you could still be limited by some extensions required)
Making sure the PHAR is runnable: the right PHP version, the right extensions
Making sure the PHAR is legitimate: if it's coming from GitHub, making sure it's downloaded via HTTPs will be enough for most of the time, an extra security step is to GPG sign the artefact and register your signature elsewhere and have a check. Some tools like phive offers this.
Updating the PHAR. Right now there is no good out-of-the box self-update command (there is some packages, but mostly unmaintained & not complete/secure enough IMO). So updating the PHAR itself is a bit inconvenient unless using a tool like phive.
Keeping a PHAR version

And regardless of how the application is installed, via Composer or as a PHAR, as mentioned above, there is an issue whenever the application will load/execute arbitrary code as there is a risk of conflicts.

In light of the above and @naderman analysis of the issue. I think a require-tool option would solve the installation issue. Be it for globally or on a project basis, what I would really love from this option is that it accepts both a regular Composer package and a PHAR:

When given a library, that it installs the library in a dedicated repository. This way no matter how many tools you install, no matter what their dependencies are, there will never be any conflict with it. My current best to go-to alternative for this right now is bamarni/composer-bin-plugin, and I think it could be a good base to this.
When installing a PHAR, it would simply download the PHAR. An existing alternative solution is tommy-muehle/composer-tool-installer-plugin.
- The current plugin however does not handle version constraints which is a bit of a shame. Maybe with the require-tool, a possible solution would be to give the regular composer package name & version constraint, but setting an option "phar" and it would download the PHAR over the package (e.g. by expecting the PHAR to be in the release besides the code source).
- Another concern is when one wishes to go an extra step security wise and GPG sign his PHAR. In this case, maybe a similar approach to phive could be picked. Actually phive has a composer plugin as well which could be used as a base.

IMO this solution would elegantly solve the issue of installing tools/apps via Composer whilst being able to keeping track of them on a project level when desired.

Now regarding the last issue of code conflict. I think indeed Humbug PHP-Scoper is the way to go (disclaimer: I am the maintainer of the tool). It is in my opinion not perfect, mostly because:

It is not battle tested on thousands of projects. That said I've been using it extensively for a lot of projects nonetheless and it works relatively well. It's however bound to have some lingering bugs (like all tools I guess).
It is heavily dependent on PHP-Parser which means if PHP-Parser is lagging behind in a PHP version support so will PHP-Scoper.
As a consequence of the point above, PHP-Parser is very slow. It's heavy AST manipulation. While there is tricks to make it go faster (e.g. leveraging parallelism or being smarter about what is scoped like done in Box), it is and will always be way to slow for allowing a on the fly package scoping integrating in the package manager like you can see for example in JavaScript with yarn or npm.
Because PHP is what it is as a language, there will always be edge cases that PHP-Scoper will not be able to take care of itself. A non exhaustive list of examples:
- A dynamic reference to a class name, e.g. $x = 'MyClass'.$foo.'Handler; or $x = preg_replace('/Foo/', 'Acme', Bar::class);
- Exotic usage of autoloading mechanism e.g. like done in some Hoa projects (IIRC hoaproject/Consistency in peculiar)
- Referencing functions relying on the autoloading fallback, i.e.:

namespace Acme;

tap(); // is it \tap() or \Acme\tap()?

- Relying on some unorthodox ways to configure your PHP application, like the Symfony YAML/XML configuration files (actually PHP-Scoper has a way to deal with is but it's a solution specifically developed for Symfony, not a generic way to tackle the issue)

PHP-Scoper offers a number of ways to overcome those limitations and you will always find a way to make it work regardless of how dirty the code is, but my point is unlike you have a very traditional standard code, it will most likely never achieve the experience of "it works" on the first try. I'm trying very hard to make it as easy as possible, but this is how the language and its limitations are.

All of this to say I don't think PHP-Scoper or any similar tool will ever be robust and reliable enough to scope packages on the fly natively by Composer. However one is definitely encourage to use to it:

Scope a library code and publish it, for example like PHPStan did with PHPStan shim
Scope the PHAR like done for PHP-Scoper itself or Box

I think another topic which needs to be discussed is the choice regarding the PHARs. Indeed at least in the solution I've mentioned above, we are talking of having a good integration for PHAR files with Composer.

Maybe another approach would be to completely forget about PHARs. If composer can install those tools as libraries, maybe it could install the shims instead. I.e. instead of having:

acme/foo published as a package
scoped PHAR provided for acme/foo in each release

or:

acme/foo published as a package
acme/foo-shim (scoped version of acme/foo) published as a regular Composer package

The scoped code could be uploaded as an artefact with the release and Composer would have a way to understand "this is a shim package, i.e. a code without conflict, pick this one for require-tool".

Indeed right now having a scoped PHAR or a separate a shim packages are both non-trivial amount of work for each maintainers. Publishing the shim package as an artefact still requires the work of making the scoping work, but would remove the burden to deal with PHARs or having a shim package to synchronize.

Greg Anderson · Answer 53 · Mon Feb 18 2019 02:39:07 GMT+0800 (China Standard Time)

I agree that it is a problem that Composer is doing double-duty as both a dependency manager and an installer. Other languages, such as Go, do these options with separate tools. The PHP community doesn't have a uniform installer. composer global require looks like such a thing, which is why so many people continue to use it. Lacking an installer provided by the PHP project itself, a safe and sanctioned way to do this in Composer 2 is important, and I am glad to see thought going into moving in this direction.

I am -1 on using PHP-Scoper / PHP-Parser in Composer due to the limitations described in the comment above. I am +1 on seeing PHP-Scoper, or equivalent function thereof, becoming a standard language feature in a future version of PHP. Under that scenario, then supporting it in a dependency manager or installer would be natural.

Théo FIDRY · Answer 54 · Mon Feb 18 2019 03:21:31 GMT+0800 (China Standard Time)

I am -1 on using PHP-Scoper / PHP-Parser in Composer due to the limitations described in the comment above. I am +1 on seeing PHP-Scoper, or equivalent function thereof, becoming a standard language feature in a future version of PHP. Under that scenario, then supporting it in a dependency manager or installer would be natural.

Just to make sure there is no confusion: I suggested a way in which it could make it slightly easier (publishing the scoped content as an artefact - it doesn't have any dependence on the tool used to achieve it neither is there any promise that the code is perfectly scoped either) but the responsibility to scope a library/application is entirely on the maintainer(s).

Jordi Boggiano · Answer 55 · Wed Mar 06 2019 15:04:21 GMT+0800 (China Standard Time)

This seems to implement exactly what some of you discussed above, have a look at it: https://github.com/tommy-muehle/tooly-composer-script

Jaap van Otterdijk · Answer 56 · Sat Oct 19 2019 19:44:11 GMT+0800 (China Standard Time)

See also the discussion in phive about this topic: phar-io/phive#88

Jordi Boggiano · Answer 57 · Tue Jan 26 2021 23:59:38 GMT+0800 (China Standard Time)

Closing in favor of #9636 which is a good summary and has less history to read through. Please subscribe there if you're still interested in this issue.