proposal: switch to using nested modules for -m mode resolution

Question

proposal: switch to using nested modules for -m mode resolution

myitcv opened this issue 5 years ago · comments

Building on the ideas started in #44.

Currently, the -m flag tells gobin to use the main module for resolution of dependencies. -m can also, therefore, be thought of as "non-global" mode. In -m mode, we use the guidance at golang/go#25922 (comment) for keeping track of the tools we require in a build-ignored tools.go file.

Under this behaviour however, -m mode ends up cluttering our main module.

#44 proposed an alternative means of capturing the list of tools we require in a go.tools file. This issue more precisely specifies the behaviour as follows:

go.tools will capture the import paths of tools we require, e.g.:

# go.tools
golang.org/x/tools/cmd/stringer
example.com/blah/v2

gobin will then use synthetic nested modules (in a .gobin directory) per main package for module resolution (including the main package's module). For example:

// $PWD/.gobin/golang.org/x/tools/cmd/stringer/go.mod
module mod

require (
    golang.org/x/tools v0.0.0-20190826060629-95c3470cfb70
)

Points to note:

The filepath of nested modules will need to use the same escaping as seen in the cmd/go module cache
All non-directory replace directives in the main package's main module will appear in the synthetic nested module for a main package

cc @rogpeppe @mvdan

Daniel Martí · Answer 1 · Mon Sep 02 2019 22:26:59 GMT+0800 (China Standard Time)

Do we need one go.mod file per tool dependency? For example, I see no value in different tools using different versions of x/tools/go/packages, and it's just going to make the builds slower and the setup more complex.

Going even further - why not add the go.mod/go.sum data right into go.tools? As a silly example, using txtar itself:

$ cat go.tools
golang.org/x/tools/cmd/stringer
example.com/blah/v2

-- go.mod --
module mod

require (
    golang.org/x/tools v0.0.0-20190826060629-95c3470cfb70
)
-- go.sum --
etc etc

Daniel Martí · Answer 2 · Mon Sep 02 2019 22:41:18 GMT+0800 (China Standard Time)

To clarify; my point about using a single go.tools file is to reduce the overhead that is adding more files or directories to the root of a module.

However, one disadvantage is that not having the plain files on disk somewhere would make it harder to use vanilla cmd/go to use or update the go.mod and go.sum files. But this is something that gobin could allow; for example:

$ gobin go mod graph
# extracts go.mod and go.sum into a temporary folder
# runs the cmd/go command in said folder (in this case 'go mod graph')
# saves go.mod and go.sum back into go.tools

Paul Jolly · Answer 3 · Mon Sep 02 2019 23:07:23 GMT+0800 (China Standard Time)

Do we need one go.mod file per tool dependency?

I would argue yes, and I suspect @rogpeppe would too.

Various reasons:

brings versioned tool use within a project (via go.tools) inline with the global use, i.e. each tool is a distinct and separate entry point
means tools' respective dependencies don't ever mix and influence each other, which ensures reproducibility over time, especially globally vs locally
allows us to make more aggressive assertions about the install target. That was the intent of #80 but precisely because the tools' dependencies interact with each other (via the main module in this instance) we can't make these assertions
allows use to share the same gobin cache target for a project's versioned (i.e. non main module and non directory-replaced) module packages, and global tools

Just to confirm, the "overhead" for modules requiring tools under this proposal would be:

a go.tools
a directory .gobin/ containing a tree of directories for each main package, each containing a go.{mod,sum} pair

Is your concern about the number of files in this .gobin directory?

Daniel Martí · Answer 4 · Mon Sep 02 2019 23:10:04 GMT+0800 (China Standard Time)

Gotcha, thanks. I hadn't thought about the side effects as much as you two have. Not convinced that this is a clear decision, but I see it's a tradeoff.

Is your concern about the number of files in this .gobin directory?

My concern is the number of files in the root of a module. Right now we already have go.mod and go.sum, and we've been thinking about go.tools for a while. .gobin/ (or whatever other directory) would be the fourth, which is already a non-trivial amount of clutter.

Paul Jolly · Answer 5 · Mon Sep 02 2019 23:15:48 GMT+0800 (China Standard Time)

Not convinced that this is a clear decision, but I see it's a tradeoff.

Agreed.

... which is already a non-trivial amount of clutter.

Indeed.

Roger Peppe · Answer 6 · Tue Sep 03 2019 07:04:53 GMT+0800 (China Standard Time)

note that you only need such a nested module for tools that don't contain a go.mod file. Hopefully that will become increasingly uncommon as time goes on.

Paul Jolly · Answer 7 · Tue Sep 03 2019 15:49:26 GMT+0800 (China Standard Time)

There are two reasons why I think we should still have a go.{mod,sum} per main package module:

handles the case where the main package's module might not be tidy/complete
allows us to bake version information into the target binary, which can then be revealed using go version -m /path/to/binary in Go 1.13. If we simply install from the main module we will only get a devel version identifier

Daniel Martí · Answer 8 · Tue Sep 03 2019 16:43:38 GMT+0800 (China Standard Time)

handles the case where the main package's module might not be tidy/complete

If we live in a world where proxy.golang.org is the standard for open source software, wouldn't (or couldn't) this already be covered?

If we simply install from the main module we will only get a devel version identifier

Are you positive about this? I understand why a go build ./some/tool doesn't contain version info at the moment, as it does no module version resolution. But surely a go install some-tool would contain the version info, as it would be pulled from the main go.mod.

Paul Jolly · Answer 9 · Tue Sep 03 2019 18:06:47 GMT+0800 (China Standard Time)

handles the case where the main package's module might not be tidy/complete

If we live in a world where proxy.golang.org is the standard for open source software, wouldn't (or couldn't) this already be covered?

I don't think so. Assume either the main module or a (transitive) dependency has incomplete dependencies. At time t1 missing dependency d1 resolves to v1.0.0 of; but some time later at t2, after v1.1.0 has been released, it resolves to v1.1.0. We don't have reproducible builds this way, and that's something I think we want and need.

If we simply install from the main module we will only get a devel version identifier

Are you positive about this? I understand why a go build ./some/tool doesn't contain version info at the moment, as it does no module version resolution. But surely a go install some-tool would contain the version info, as it would be pulled from the main go.mod.

But the go.mod itself has no version information. Perhaps I not quite getting what you're referring to; could you explain further with a cmd/go "equivalent" perhaps? i.e. the steps that would in effect be followed by gobin?