Referencing packages using path type
eithed opened this issue · comments
Using Satis with following satis.json
{
"name": "eithed/satis",
"homepage": "https://eithed.github.io/",
"repositories": [{
"packagist.org": false
},
{
"type": "path",
"url": "a"
},
{
"type": "path",
"url": "b"
}],
"require-all": true
}
and following file structure
├── a
│ └── composer.json
└── b
└── composer.json
Satis correctly generates packages.json with include/all$.json containing:
{
"packages": {
"eithed/a": {
"dev-master": {
"name": "eithed/a",
"version": "dev-master",
"version_normalized": "9999999-dev",
"dist": {
"type": "path",
"url": "a",
"reference": "886606395a3dd35bab7a4ccfdddf606ceb4465f7"
},
"type": "library"
}
},
"eithed/b": {
"dev-master": {
"name": "eithed/b",
"version": "dev-master",
"version_normalized": "9999999-dev",
"dist": {
"type": "path",
"url": "b",
"reference": "055ad080da1f5d2924d6c04d8ea5f4c45e1d1c55"
},
"type": "library"
}
}
}
}
However, when using it, for composer.json:
{
"name": "eithed/subrepo-a",
"require": {
"eithed/a": "dev-master"
},
"repositories": [
{
"type": "composer",
"url": "https://eithed.github.io"
}
]
}I get the following output:
Source path "a" is not found for package eithed/a
The intent here is to have a monorepo with multiple packages; if I'll need to use one of the packages within separate project I'd like to simply reference it as a composer repo and require the package I'm interested in. Is this possible at all?
Missing a stacktrace.
My opinion is that a monorepo and Composer don't work together very well. Composer sees the world as "one repo is one package", and this repo provides it's Composer metadata as composer.json in the top-level directory, as well as get's tagged with versions that exclusively apply to only that package.
Having two or more packages inside a repo is usually hard, because you cannot tag them individually. Looking around, you'll quickly find Symfony, and they are using that monorepo approach for their development of all components, but then break apart each component individually as a git subtree export. Each new tag in the main repo is duplicated to each component, so they are all getting tagged even if they did not change at all. Composer and Satis do not have an issue with the result, as each component has it's individual repo, and they do not care how these were created. And the tagging of unchanged components isn't even necessary for the consumers, but probably useful for maintaining purposes: A bugreport would always be able to state the latest tagged version and thus remove any doubts about the reporter using an outdated component version.
Re-reading your intent:
The intent here is to have a monorepo with multiple packages; if I'll need to use one of the packages within separate project I'd like to simply reference it as a composer repo and require the package I'm interested in. Is this possible at all?
I'm tempted to answer "no". You'd always clone the whole monorepo in order to get access to the component you are interested in. But then you could also treat that whole monorepo as a single package and be done without any problems. Single version tag for everything inside, single URL path, no file path.
Alternatively you could provide ZIP packages (either by explicitly creating them or having some kind of on-demand automagic that creates them - just like the Github ZIP download that is utilized by Composer) that contain only the relevant part of that monorepo, and make this information available through Satis. This would enable you to download this ZIP to install only that part in another project. But this still has the same effects of having a monorepo and splitting it up into single repos. You'll deal with different versions that have been tagged, alternatively you'll deal with depending on a branch that has no versions (hint: you don't want to do that, it's like removing all the good parts from having multiple repos, and also removing all the good parts from using a monorepo).
@alcohol Please find attached
Exception trace:
() at phar:///usr/local/bin/composer/src/Composer/Downloader/PathDownloader.php:45
Composer\Downloader\PathDownloader->download() at phar:///usr/local/bin/composer/src/Composer/Downloader/DownloadManager.php:216
Composer\Downloader\DownloadManager->download() at phar:///usr/local/bin/composer/src/Composer/Installer/LibraryInstaller.php:197
Composer\Installer\LibraryInstaller->installCode() at phar:///usr/local/bin/composer/src/Composer/Installer/LibraryInstaller.php:101
Composer\Installer\LibraryInstaller->install() at phar:///usr/local/bin/composer/src/Composer/Installer/InstallationManager.php:173
Composer\Installer\InstallationManager->install() at phar:///usr/local/bin/composer/src/Composer/Installer/InstallationManager.php:160
Composer\Installer\InstallationManager->execute() at phar:///usr/local/bin/composer/src/Composer/Installer.php:611
Composer\Installer->doInstall() at phar:///usr/local/bin/composer/src/Composer/Installer.php:232
Composer\Installer->run() at phar:///usr/local/bin/composer/src/Composer/Command/InstallCommand.php:122
Composer\Command\InstallCommand->execute() at phar:///usr/local/bin/composer/vendor/symfony/console/Command/Command.php:245
Symfony\Component\Console\Command\Command->run() at phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:835
Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:185
Symfony\Component\Console\Application->doRun() at phar:///usr/local/bin/composer/src/Composer/Console/Application.php:281
Composer\Console\Application->doRun() at phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:117
Symfony\Component\Console\Application->run() at phar:///usr/local/bin/composer/src/Composer/Console/Application.php:113
Composer\Console\Application->run() at phar:///usr/local/bin/composer/bin/composer:61
require() at /usr/local/bin/composer:24
Are you using absolute or relative paths? Cause if they are relative, composer will try to use them relatively from the project you are trying to install your dependencies in, not from your satis source.
@SvenRtbg I'm currently researching which approach to use - subtree vs. monorepo. While this functionality is not a stopper with the monorepo approach (at the moment - all packages will be within monorepo), I'd want to make sure of the caveats.
Currently, what I'm finding with subtree is that when working on functionality that touches multiple sub repos pushing upstream is pretty cumbersome (also I wish there was a command to git pull subtree all). If you know how development process at Symfony looks like when working on functionality that spans multiple repos, I'd appreciate the link.
Regarding the zip - yup, that's what I was thinking as well. It's always an alternative - not great, but it is.
@alcohol I don't know if I understand the question. In subrepo-a I'm referencing an absolute URL. In Satis I'm using relative paths. Should I be doing this instead?
{
"name": "eithed/satis",
"homepage": "https://eithed.github.io/",
"repositories": [{
"packagist.org": false
},
{
"type": "path",
"url": "https://eithed.github.io/a"
},
{
"type": "path",
"url": "https://eithed.github.io/b"
}],
"require-all": true
}
Path can't point to a url. It has to be on the filesystem.
Satis has to be on the same filesystem/machine as where you want to install your dependencies.
@alcohol I have hoped that if I'm pointing to Satis as so:
{
"name": "eithed/subrepo-a",
"require": {
"eithed/a": "dev-master"
},
"repositories": [
{
"type": "composer",
"url": "https://eithed.github.io"
}
]
}
with https://eithed.github.io containing my Satis and https://eithed.github.io/a/ being a package, then that package would get downloaded using path.
Given that I have Satis on my local, how would I reference it in the json above?
You've lost me. I'm not entirely sure what you are trying to achieve, but I get the impression that it is not possible.
@alcohol I'm trying to have multiple packages within one repository. If I have a structure:
├── a
│ └── composer.json
└── b
└── composer.json
with Satis living in the root and referencing packages a and b, I would be able to, within my other repo specify that I want to use Satis repository and then require one of the packages exposed this way
I'd support this view.
Arguing from the end: How would Composer pull the dependency in a general case? Like in "a new machine from a new developer who tries to composer install"? If there's a ZIP with just the code for that part, it'll work. Alternatively, there would be a dedicated (subtree) repo for just that part that may be cloned.
How to create that?
Satis cannot apply some magic to make Composer work differently. All it may do is create the pointers to something differently - however the thing found there has to be something Composer can work with, and the pointer should also work in a general case. Local relative paths that are only populated and work if some arbitrary setup procedures have been followed are not a good idea (from a general view point) and not explicitly supported.
To get yourself a solution, try to get to a situation where Composer would work out of the box.
There are not that many obvious results for the Symfony subtree process. I found this blogpost from Fabien https://symfony.com/blog/symfony2-components-as-standalone-packages and this Stackoverflow question referring to exactly that blog post, but it has more details how it is being done.
Note that it is some effort, and I personally would rather give up on the monorepo idea instead, splitting the reusable components into dedicated repos. Because that's what you'd do anyways when following the Symfony approach. But in the end Satis and Composer would be working with single repos just fine.
Edit:
Your approach seems to be that you think having a satis repo with subcomponents will magically download the whole repo. It does not work that way. Satis creates static files with meta data, and Composer will usually only download (via network connection) the files it absolutely requires. Composer will not download "the whole Satis", you'd have to clone that repo before in a manual step. But even then it sounds like a bad idea, because you'd either have to force a single copy of that satis monorepo, or duplicate that. And duplication is required when using more than one machine.
Thank you very much - I have indeed hoped that Satis will point to both packages allowing them to be referenced, as long as they live within their own separate folders and have composer.json providing necessary metadata.
Each package should live in its own repository. What you want to do is not possible.
Regarding Symfony subtree approach, https://www.subtreesplit.com/ could be a good place to look at.
Or if you don't want to use a third party service, have a look at https://github.com/splitsh/lite
Will do! Thank you very much for your help!