cmd/go: 'go install' should install executables in module mode outside a module

Question

cmd/go: 'go install' should install executables in module mode outside a module

jayconrod opened this issue 4 years ago · comments

Jay Conrod commented 4 years ago

Authors: Jay Conrod, Daniel Martí

Last Updated: 2020-09-29

Design doc: CL 243077
Comments on the CL are preferred over comments on this issue.

Abstract

Authors of executables need a simple, reliable, consistent way for users to
build and install exectuables in module mode without updating module
requirements in the current module's go.mod file.

Background

go get is used to download and install executables, but it's also responsible
for managing dependencies in go.mod files. This causes confusion and
unintended side effects: for example, the command
go get golang.org/x/tools/gopls builds and installs gopls. If there's a
go.mod file in the current directory or any parent, this command also adds a
requirement on the module golang.org/x/tools/gopls, which is usually not
intended. When GO111MODULE is not set, go get will also run in GOPATH mode
when invoked outside a module.

These problems lead authors to write complex installation commands such as:

(cd $(mktemp -d); GO111MODULE=on go get golang.org/x/tools/gopls)

Proposal

We propose augmenting the go install command to build and install packages
at specific versions, regardless of the current module context.

go install golang.org/x/tools/gopls@v0.4.4

To eliminate redundancy and confusion, we also propose deprecating and removing
go get functionality for building and installing packages.

Details

The new go install behavior will be enabled when an argument has a version
suffix like @latest or @v1.5.2. Currently, go install does not allow
version suffixes. When a version suffix is used:

go install runs in module mode, regardless of whether a go.mod file is
present. If GO111MODULE=off, go install reports an error, similar to
what go mod download and other module commands do.
go install acts as if no go.mod file is present in the current directory
or parent directory.
No module will be considered the "main" module.
Errors are reported in some cases to ensure that consistent versions of
dependencies are used by users and module authors. See Rationale below.
- Command line arguments must not be meta-patterns (all, std, cmd)
  or local directories (./foo, /tmp/bar).
- Command line arguments must refer to main packages (executables). If a
  argument has a wildcard (...), it will only match main packages.
- Command line arguments must refer to packages in one module at a specific
  version. All version suffixes must be identical. The versions of the
  installed packages' dependencies are determined by that module's go.mod
  file (if it has one).
- If that module has a go.mod file, it must not contain directives that
  would cause it to be interpreted differently if the module were the main
  module. In particular, it must not contain replace or exclude
  directives.

If go install has arguments without version suffixes, its behavior will not
change. It will operate in the context of the main module. If run in module mode
outside of a module, go install will report an error.

With these restrictions, users can install executables using consistent commands.
Authors can provide simple installation instructions without worrying about
the user's working directory.

With this change, go install would overlap with go get even more, so we also
propose deprecating and removing the ability for go get to install packages.

In Go 1.16, when go get is invoked outside a module or when go get is
invoked without the -d flag with arguments matching one or more main
packages, go get would print a deprecation warning recommending an
equivalent go install command.
In a later release (likely Go 1.17), go get would no longer build or install
packages. The -d flag would be enabled by default. Setting -d=false would
be an error. If go get is invoked outside a module, it would print an error
recommending an equivalent go install command.

Examples

# Install a single executable at the latest version
$ go install example.com/cmd/tool@latest

# Install multiple executables at the latest version
$ go install example.com/cmd/...@latest

# Install at a specific version
$ go install example.com/cmd/tool@v1.4.2

Current `go install` and `go get` functionality

go install is used for building and installing packages within the context of
the main module. go install reports an error when invoked outside of a module
or when given arguments with version queries like @latest.

go get is used both for updating module dependencies in go.mod and for
building and installing executables. go get also works differently depending
on whether it's invoked inside or outside of a module.

These overlapping responsibilities lead to confusion. Ideally, we would have one
command (go install) for installing executables and one command (go get) for
changing dependencies.

Currently, when go get is invoked outside a module in module mode (with
GO111MODULE=on), its primary purpose is to build and install executables. In
this configuration, there is no main module, even if only one module provides
packages named on the command line. The build list (the set of module versions
used in the build) is calculated from requirements in go.mod files of modules
providing packages named on the command line. replace or exclude directives
from all modules are ignored. Vendor directories are also ignored.

When go get is invoked inside a module, its primary purpose is to update
requirements in go.mod. The -d flag is often used, which instructs go get
not to build or install packages. Explicit go build or go install commands
are often better for installing tools when dependency versions are specified in
go.mod and no update is desired. Like other build commands, go get loads the
build list from the main module's go.mod file, applying any replace or
exclude directives it finds there. replace and exclude directives in other
modules' go.mod files are never applied. Vendor directories in the main module
and in other modules are ignored; the -mod=vendor flag is not allowed.

The motivation for the current go get behavior was to make usage in module
mode similar to usage in GOPATH mode. In GOPATH mode, go get would download
repositories for any missing packages into $GOPATH/src, then build and install
those packages into $GOPATH/bin or $GOPATH/pkg. go get -u would update
repositories to their latest versions. go get -d would download repositories
without building packages. In module mode, go get works with requirements in
go.mod instead of repositories in $GOPATH/src.

Rationale

Why can't `go get` clone a git repository and build from there?

In module mode, the go command typically fetches dependencies from a
proxy. Modules are distributed as zip files that contain sources for specific
module versions. Even when go connects directly to a repository instead of a
proxy, it still generates zip files so that builds work consistently no matter
how modules are fetched. Those zip files don't contain nested modules or vendor
directories.

If go get cloned repositories, it would work very differently from other build
commands. That causes several problems:

It adds complication (and bugs!) to the go command to support a new build
mode.
It creates work for authors, who would need to ensure their programs can be
built with both go get and go install.
It reduces speed and reliability for users. Modules may be available on a
proxy when the original repository is unavailable. Fetching modules from a
proxy is roughly 5-7x faster than cloning git repositories.

Why can't vendor directories be used?

Vendor directories are not included in module zip files. Since they're not
present when a module is downloaded, there's no way to build with them.

We don't plan to include vendor directories in zip files in the future
either. Changing the set of files included in module zip files would break
go.sum hashes.

Why can't directory `replace` directives be used?

For example:

replace example.com/sibling => ../sibling

replace directives with a directory path on the right side can't be used
because the directory must be outside the module. These directories can't be
present when the module is downloaded, so there's no way to build with them.

Why can't module `replace` directives be used?

For example:

replace example.com/mod v1.0.0 => example.com/fork v1.0.1-bugfix

It is technically possible to apply these directives. If we did this, we would
still want some restrictions. First, an error would be reported if more than one
module provided packages named on the command line: we must be able to identify
a main module. Second, an error would be reported if any directory replace
directives were present: we don't want to introduce a new configuration where
some replace directives are applied but others are silently ignored.

However, there are two reasons to avoid applying replace directives at all.

First, applying replace directives would create inconsistency for users inside
and outside a module. When a package is built within a module with go build or
go install, only replace directives from the main module are applied, not
the module providing the package. When a package is built outside a module with
go get, no replace directives are applied. If go install applied replace
directives from the module providing the package, it would not be consistent
with the current behavior of any other build command. To eliminate confusion
about whether replace directives are applied, we propose that go install
reports errors when encountering them.

Second, if go install applied replace directives, it would take power away
from developers that depend on modules that provide tools. For example, suppose
the author of a popular code generation tool gogen forks a dependency
genutil to add a feature. They add a replace directive pointing to their
fork of genutil while waiting for a PR to merge. A user of gogen wants to
track the version they use in their go.mod file to ensure everyone on their
team uses a consistent version. Unfortunately, they can no longer build gogen
with go install because the replace is ignored. The author of gogen might
instruct their users to build with go install, but then users can't track the
dependency in their go.mod file, and they can't apply their own require and
replace directives to upgrade or fix other transitive dependencies. The author
of gogen could also instruct their users to copy the replace directive, but
this may conflict with other require and replace directives, and it may
cause similar problems for users further downstream.

Why report errors instead of ignoring `replace`?

If go install ignored replace directives, it would be consistent with the
current behavior of go get when invoked outside a module. However, in
#30515 and related discussions, we found that
many developers are surprised by that behavior.

It seems better to be explicit that replace directives are only applied
locally within a module during development and not when users build packages
from outside the module. We'd like to encourage module authors to release
versions of their modules that don't rely on replace directives so that users
in other modules may depend on them easily.

If this behavior turns out not to be suitable (for example, authors prefer to
keep replace directives in go.mod at release versions and understand that
they won't affect users), then we could start ignoring replace directives in
the future, matching current go get behavior.

Should `go.sum` files be checked?

Because there is no main module, go install will not use a go.sum file to
authenticate any downloaded module or go.mod file. The go command will still
use the checksum database (sum.golang.org) to
authenticate downloads, subject to privacy settings. This is consistent with the
current behavior of go get: when invoked outside a module, no go.sum file is
used.

The new go install command requires that only one module may provide packages
named on the command line, so it may be logical to use that module's go.sum
file to verify downloads. This avoids a problem in
#28802, a related proposal to verify downloads
against all go.sum files in dependencies: the build can't be broken by one bad
go.sum file in a dependency.

However, using the go.sum from the module named on the command line only
provides a marginal security benefit: it lets us authenticate private module
dependencies (those not available to the checksum database) when the module on
the command line is public. If the module named on the command line is private
or if the checksum database isn't used, then we can't authenticate the download
of its content (including the go.sum file), and we must trust the proxy. If
all dependencies are public, we can authenticate all downloads without go.sum.

Why require a version suffix when outside a module?

If no version suffix were required when go install is invoked outside a
module, then the meaning of the command would depend on whether the user's
working directory is inside a module. For example:

go install golang.org/x/tools/gopls

When invoked outside of a module, this command would run in GOPATH mode,
unless GO111MODULE=on is set. In module mode, it would install the latest
version of the executable.

When invoked inside a module, this command would use the main module's go.mod
file to determine the versions of the modules needed to build the package.

We currently have a similar problem with go get. Requiring the version suffix
makes the meaning of a go install command unambiguous.

Why not a `-g` flag instead of `@latest`?

To install the latest version of an executable, the two commands below would be
equivalent:

go install -g golang.org/x/tools/gopls
go install golang.org/x/tools/gopls@latest

The -g flag has the advantage of being shorter for a common use case. However,
it would only be useful when installing the latest version of a package, since
-g would be implied by any version suffix.

The @latest suffix is clearer, and it implies that the command is
time-dependent and not reproducible. We prefer it for those reasons.

Compatibility

The go install part of this proposal only applies to commands with version
suffixes on each argument. go install reports an error for these, and this
proposal does not recommend changing other functionality of go install, so
that part of the proposal is backward compatible.

The go get part of this proposal recommends deprecating and removing
functionality, so it's certainly not backward compatible. go get -d commands
will continue to work without modification though, and eventually, the -d flag
can be dropped.

Parts of this proposal are more strict than is technically necessary (for
example, requiring one module, forbidding replace directives). We could relax
these restrictions without breaking compatibility in the future if it seems
expedient. It would be much harder to add restrictions later.

Implementation

An initial implementation of this feature was merged in
CL 254365. Please try it
out!

Future directions

The behavior with respect to replace directives was discussed extensively
before this proposal was written. There are three potential behaviors:

Ignore replace directives in all modules. This would be consistent with
other module-aware commands, which only apply replace directives from the
main module (defined in the current directory or a parent directory).
go install pkg@version ignores the current directory and any go.mod
file that might be present, so there is no main module.
Ensure only one module provides packages named on the command line, and
treat that module as the main module, applying its module replace
directives from it. Report errors for directory replace directives. This
is feasible, but it may have wider ecosystem effects; see "Why can't module
replace directives be used?" above.
Ensure only one module provides packages named on the command line, and
report errors for any replace directives it contains. This is the behavior
currently proposed.

Most people involved in this discussion have advocated for either (1) or (2).
The behavior in (3) is a compromise. If we find that the behavior in (1) is
strictly better than (2) or vice versa, we can switch to that behavior from
(3) without an incompatible change. Additionally, (3) eliminates
ambiguity about whether replace directives are applied for users and module
authors.

Note that applying directory replace directives is not considered here for
the reasons in "Why can't directory replace directives be used?".

Appendix: FAQ

Why not apply `replace` directives from all modules?

In short, replace directives from different modules would conflict, and
that would make dependency management harder for most users.

For example, consider a case where two dependencies replace the same module
with different forks.

// in example.com/mod/a
replace example.com/mod/c => example.com/fork-a/c v1.0.0

// in example.com/mod/b
replace example.com/mod/c => example.com/fork-b/c v1.0.0

Another conflict would occur where two dependencies pin different versions
of the same module.

// in example.com/mod/a
replace example.com/mod/c => example.com/mod/c v1.1.0

// in example.com/mod/b
replace example.com/mod/c => example.com/mod/c v1.2.0

To avoid the possibility of conflict, the go command ignores replace
directives in modules other than the main module.

Modules are intended to scale to a large ecosystem, and in order for upgrades
to be safe, fast, and predictable, some rules must be followed, like semantic
versioning and import compatibility.
Not relying on replace is one of these rules.

How can module authors avoid `replace`?

replace is useful in several situations for local or short-term development,
for example:

Changing multiple modules concurrently.
Using a short-term fork of a dependency until a change is merged upstream.
Using an old version of a dependency because a new version is broken.
Working around migration problems, like golang.org/x/lint imported as
github.com/golang/lint. Many of these problems should be fixed by lazy
module loading (#36460).

replace is safe to use in a module that is not depended on by other modules.
It's also safe to use in revisions that aren't depended on by other modules.

If a replace directive is just meant for temporary local development by one
person, avoid checking it in. The -modfile flag may be used to build with
an alternative go.mod file. See also
#26640 a feature request for a
go.mod.local file containing replacements and other local modifications.
If a replace directive must be checked in to fix a short-term problem,
ensure at least one release or pre-release version is tagged before checking
it in. Don't tag a new release version with replace checked in (pre-release
versions may be okay, depending on how they're used). When the go command
looks for a new version of a module (for example, when running go get with
no version specified), it will prefer release versions. Tagging versions lets
you continue development on the main branch without worrying about users
fetching arbitrary commits.
If a replace directive must be checked in to solve a long-term problem,
consider solutions that won't cause issues for dependent modules. If possible,
tag versions on a release branch with replace directives removed.

When would `go install` be reproducible?

The new go install command will build an executable with the same set of
module versions on every invocation if both the following conditions are true:

A specific version is requested in the command line argument, for example,
go install example.com/cmd/foo@v1.0.0.
Every package needed to build the executable is provided by a module required
directly or indirectly by the go.mod file of the module providing the
executable. If the executable only imports standard library packages or
packages from its own module, no go.mod file is necessary.

An executable may not be bit-for-bit reproducible for other reasons. Debugging
information will include system paths (unless -trimpath is used). A package
may import different packages on different platforms (or may not build at all).
The installed Go version and the C toolchain may also affect binary
reproducibility.

What happens if a module depends on a newer version of itself?

go install will report an error, as go get already does.

This sometimes happens when two modules depend on each other, and releases
are not tagged on the main branch. A command like go get example.com/m@master
will resolve @master to a pseudo-version lower than any release version.
The go.mod file at that pseudo-version may transitively depend on a newer
release version.

go get reports an error in this situation. In general, go get reports
an error when command line arguments different versions of the same module,
directly or indirectly. go install doesn't support this yet, but this should
be one of the conditions checked when running with version suffix arguments.

Appendix: usage of replace directives

In this proposal, go install would report errors for replace directives in
the module providing packages named on the command line. go get ignores these,
but the behavior may still surprise module authors and users. I've tried to
estimate the impact on the existing set of open source modules.

I started with a list of 359,040 main packages that Russ Cox built during an
earlier study.
I excluded packages with paths that indicate they were homework, examples,
tests, or experiments. 187,805 packages remained.
Of these, I took a random sample of 19,000 packages (about 10%).
These belonged to 13,874 modules. For each module, I downloaded the "latest"
version go get would fetch.
I discarded repositories that were forks or couldn't be retrieved. 10,618
modules were left.
I discarded modules that didn't have a go.mod file. 4,519 were left.
Of these:
- 3982 (88%) don't use replace at all.
- 71 (2%) use directory replace only.
- 439 (9%) use module replace only.
- 27 (1%) use both.
- In the set of 439 go.mod files using module replace only, I tried to
  classify why replace was used. A module may have multiple replace
  directives and multiple classifications, so the percentages below don't add
  to 100%.
- 165 used replace as a soft fork, for example, to point to a bug fix PR
  instead of the original module.
- 242 used replace to pin a specific version of a dependency (the module
  path is the same on both sides).
- 77 used replace to rename a dependency that was imported with another
  name, for example, replacing github.com/golang/lint with the correct path,
  golang.org/x/lint.
- 30 used replace to rename golang.org/x repos with their
  github.com/golang mirrors.
- 11 used replace to bypass semantic import versioning.
- 167 used replace with k8s.io modules. Kubernetes has used replace to
  bypass MVS, and dependent modules have been forced to do the same.
- 111 modules contained replace directives I couldn't automatically
  classify. The ones I looked at seemed to mostly be forks or pins.

The modules I'm most concerned about are those that use replace as a soft fork
while submitting a bug fix to an upstream module; other problems have other
solutions that I don't think we need to design for here. Modules using soft fork
replacements are about 4% of the the modules with go.mod files I sampled (165
/ 4519). This is a small enough set that I think we should move forward with the
proposal above.

GopherBot · Answer 1 · Sat Jul 18 2020 01:52:56 GMT+0800 (China Standard Time)

Change https://golang.org/cl/243077 mentions this issue: design: add 40276-go-get-b.md

Patrick Baxter · Answer 2 · Sat Jul 18 2020 03:01:35 GMT+0800 (China Standard Time)

Thanks for the proposal!

go get acts as if no go.mod file is present in the current directory or
parent directory.

No module will be considered the "main" module.

Would it also be accurate to say that the go.mod being used as the main module is that of the module being built rather than no "main" module? Erroring on replaces is then simply a consequence of the implementation preventing replaces from being respected because we consider the module the "main". If the intention was not to have a main module, one might expect replaces to be ignored just like go get does today outside of a module. Regardless on how go get -b and go get outside a module end up unifying, it seems that the intention of go get -b today is to build reproducible binaries using the packages module as the main module or fail if unable to do so. I know the outcome is the same here, but I think the stated intention helps a user know what to expect of this mode.

The -u flag may be used together with -b. As usual, -u upgrades modules
providing packages imported directly or indirectly by packages named on the
command line

go get is used to download and install executables, but it's also responsible
for managing dependencies in go.mod files. This causes confusion and
unintended side effects

Since go get -b is a distinct mode of go get that is explicitly is not for managing dependencies of a local go.mod, my instinct here is that -u shouldn't be an option as it further confuses the distinction between using go get for building binaries vs managing dependencies. I at least don't know a scenario where I would want to build a project with updated dependencies and not view and work with the go.mod afterword. For something like that I would always have the repo cloned already as I am now building previously unused combination of dependencies with that project and would probably also want to run tests and maybe downgrade certain dependencies if necessary.

Jay Conrod · Answer 3 · Sat Jul 18 2020 06:28:40 GMT+0800 (China Standard Time)

Would it also be accurate to say that the go.mod being used as the main module is that of the module being built rather than no "main" module? Erroring on replaces is then simply a consequence of the implementation preventing replaces from being respected because we consider the module the "main". If the intention was not to have a main module, one might expect replaces to be ignored just like go get does today outside of a module. Regardless on how go get -b and go get outside a module end up unifying, it seems that the intention of go get -b today is to build reproducible binaries using the packages module as the main module or fail if unable to do so. I know the outcome is the same here, but I think the stated intention helps a user know what to expect of this mode.

That's not quite the intent: there will actually be no main module.

In an earlier iteration on #30515, I suggested that go get -b would act the same as go get, but it would ignore the go.mod in the current directory: it would ignore replace. Many people found that unintuitive though. It seemed like we eventually had consensus that go get -b should report errors if there's more than one module on the command line or if there are replace directives. That way, you'll build the same binary whether it's: using go build inside the module providing the executable, using go get outside any module, or using go get -b anywhere.

Since go get -b is a distinct mode of go get that is explicitly is not for managing dependencies of a local go.mod, my instinct here is that -u shouldn't be an option as it further confuses the distinction between using go get for building binaries vs managing dependencies. I at least don't know a scenario where I would want to build a project with updated dependencies and not view and work with the go.mod afterword. For something like that I would always have the repo cloned already as I am now building previously unused combination of dependencies with that project and would probably also want to run tests and maybe downgrade certain dependencies if necessary.

There is some use for -u here: it lets you build and install an executable with updated dependencies. go get -u can already do that when run outside a module. I don't expect it's very common, but we get it for free, so I don't think there's a good reason to break it.

Patrick Baxter · Answer 4 · Sat Jul 18 2020 07:45:03 GMT+0800 (China Standard Time)

had consensus that go get -b should report errors if there's more than one module on the command line or if there are replace directives. That way, you'll build the same binary whether it's: using go build inside the module providing the executable, using go get outside any module, or using go get -b anywhere.

I think we are on the same page here then regardless!

Jay Conrod · Answer 5 · Tue Aug 11 2020 05:46:02 GMT+0800 (China Standard Time)

Based on discussion in https://groups.google.com/g/golang-tools/c/BRCgqwWLwoY/m/pKuttL9cAwAJ and on Slack, I'd like to take this in a different direction, so I've updated CL 243077 and the copy above.

Instead of adding go get -b, go install would have the proposed functionality when invoked with arguments with a version suffix. For example the command below would install gopls v0.4.4. It could be run from any directory and would ignore the module in the current directory if there is one.

go install golang.org/x/tools/gopls@v0.4.4

go get would no longer build or install packages. It would only be used for changing dependencies.

PTAL and comment on CL 243077 if you have any thoughts. This would be a significant change, but it would be good to have a clear separation of responsibility for go get and go install.

GopherBot · Answer 6 · Sat Sep 12 2020 00:48:29 GMT+0800 (China Standard Time)

Change https://golang.org/cl/254365 mentions this issue: cmd/go: implement 'go install pkg@version'

Andy Bursavich · Answer 7 · Tue Sep 22 2020 05:08:48 GMT+0800 (China Standard Time)

@jayconrod

Moving conversation over from #40728, my only sticking point is that I agree with you here: #30515 (comment) 💯

When called from inside a module, the current behavior in tip is great when referring to a module-local package path, but when referring to a non-local package path go install package should behave the same as go install package@latest, so as to be consistent with other module-related commands.

Jay Conrod · Answer 8 · Tue Sep 22 2020 22:02:46 GMT+0800 (China Standard Time)

@abursavich There are a couple significant problems with that approach. It would be a break from current go install behavior, and it would be inconsistent with other build commands (go build, go run, and so on).

It is confusing that there will be two variants of go install, but instead of changing the semantics of go install pkg, I think the best way to resolve that would be to remove go install pkg and only allow go install pkg@version. That seems like too disruptive of a change to me, but I'm curious to hear other opinions.

Andy Bursavich · Answer 9 · Tue Sep 22 2020 23:13:04 GMT+0800 (China Standard Time)

I think it's important to keep go install pkg for module-local packages, particularly since replace directives are going to be ignored with @version... If you really want to drop the consistency hammer, go get should require an explicit @version to add or update a specific package in module-aware mode.

Andy Bursavich · Answer 10 · Tue Sep 22 2020 23:28:38 GMT+0800 (China Standard Time)

I'm just gonna withdraw my comments and step out of the way of progress. Leave it like it is in tip :)

Alessandro Arzilli · Answer 11 · Thu Sep 24 2020 04:13:44 GMT+0800 (China Standard Time)

Is it too late to roll back the whole "global install command" thing completely? Because I feel like the way it's being impelmented I'd rather not have it in the ecosystem at all. I think the original impetus behind it was that approximately everyone had a README file that said "use go get to install this" and in modules mode it just did a bunch of weird things.

But at this point we've lived without it for 2 years, READMEs have been changed, the community has adapted, do we still need it? And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

A side effect of the way it's being implemented is that it discourages applications from using replace directives, heavily, if it gets adopted by the community as the default way to install applications. It's strange that replace directives are in the language but the tooling discourages from using them. If this is the design we settle in then replace directives should get a deprecation warning in the documentation. But IMHO replace directives are in fact useful, more useful than a global install command, so if they can't be accomodated maybe it's the global install command that should go away.

Jay Conrod · Answer 12 · Thu Sep 24 2020 22:29:34 GMT+0800 (China Standard Time)

Is it too late to roll back the whole "global install command" thing completely?

This proposal hasn't been formally accepted yet. However, I think there's a pretty firm consensus to move forward with this in 1.16. The lack of a global install command has been a significant usability problem, and many people have asked for something that solves this problem.

But at this point we've lived without it for 2 years, READMEs have been changed, the community has adapted, do we still need it?

I don't think the community has adapted. Many instructions I've seen recommend changing to a directory outside a module, then running go get in either GOPATH mode or with an explicit GO111MODULE=on. I think we need a more convenient, consistent way of installing tools.

And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

In module mode, this would be unexpected behavior. The go command usually fetches dependencies from a proxy. Even in direct mode, it doesn't fetch a whole repository. This is discussed in the design doc.

A side effect of the way it's being implemented is that it discourages applications from using replace directives, heavily, if it gets adopted by the community as the default way to install applications. It's strange that replace directives are in the language but the tooling discourages from using them. If this is the design we settle in then replace directives should get a deprecation warning in the documentation.

replace is not deprecated, and there's no issue with using it during development.

replace is discouraged in release versions that other modules may depend on. replace directives are only applied in the main module; they're ignored by go install and go get in another module or outside modules. This leads to subtle differences in selected versions depending on what command was issued where. We felt that reporting an error was preferable to that ambiguity.

This is also discussed in the design doc and at length in the previous discussion in #30515.

Patrick Baxter · Answer 13 · Thu Sep 24 2020 23:13:43 GMT+0800 (China Standard Time)

But at this point we've lived without it for 2 years, READMEs have been changed, the community has adapted, do we still need it?

I agree with jayconrad that people haven't really adapted and switching to modules has made things more confusing between go install and go get so I think this is a big win in tooling clarity.

And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

I also think that anything that doesn't build the same binary as git clone && go build is not whats expected. However a really important part of this change is that now go install will refuse to build a module with replace directives and error out rather then ignoring it. Building with replace directives is intuitively what I want with go install but erroring on encountering them is the right thing to do instead. Related to this limitation, go install will be much faster then git clone && go build. Before this change go get would ignore replaces building external binaries so I think this change brings you closer to what you want.

Daniel Martí · Answer 14 · Thu Sep 24 2020 23:22:28 GMT+0800 (China Standard Time)

I want to give extra emphasis to a disadvantage of git clone && [...] that @jayconrod already mentioned. Using the VCS repository directly is:

slower, since you have to download a lot more (module zips are generally much smaller)
less reliable, since you can't make use of module proxies
not future-proof, since a module might change its VCS repo at any point without changing its module path

Alessandro Arzilli · Answer 15 · Thu Sep 24 2020 23:34:20 GMT+0800 (China Standard Time)

Is it too late to roll back the whole "global install command" thing completely?

This proposal hasn't been formally accepted yet. However, I think there's a pretty firm consensus to move forward with this in 1.16. The lack of a global install command has been a significant usability problem, and many people have asked for something that solves this problem.

I don't disagree with this, in fact I would also like a global install command. The problem that I have is that I dislike the way this is being implemented more than I would like having a global install.

And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

In module mode, this would be unexpected behavior. The go command usually fetches dependencies from a proxy. Even in direct mode, it doesn't fetch a whole repository. This is discussed in the design doc.

I disagree, I think that this would be the expected behavior and that the current behavior of the go command is actually a regression. go get used to do a repository clone.

Address the "Why can't go get clone a git repository and build from there?" list of problems:

It adds complication (and bugs!) to the go command to support a new build mode.

It wouldn't be a new build mode, it would be the same build mode as executing a go build in the root directory of the project. It would add an alternate way to download projects (which used to exist previously).

It creates work for authors, who would need to ensure their programs can be built with both go get and go install.

In reality most code is developed by running go build on a cloned is the normal, inevitable, way of doing development. What adds work for authors is introducing a new way of downloading the code that ignores replace directives and vendor directories. The solution is to also fix go get to build executables as if git clone && go build.

It reduces speed and reliability for users. Modules may be available on a proxy when the original repository is unavailable. Fetching modules from a proxy is roughly 5-7x faster than cloning git repositories.

I concede that this is a valid point, however correctness is more important than performance.

Speaking of unexpected behavior, I think not respecting replace directives is a far more unexpected behavior than the changing details of how a module is downloaded from the internet.

A side effect of the way it's being implemented is that it discourages applications from using replace directives, heavily, if it gets adopted by the community as the default way to install applications. It's strange that replace directives are in the language but the tooling discourages from using them. If this is the design we settle in then replace directives should get a deprecation warning in the documentation.

replace is not deprecated, and there's no issue with using it during development.

replace is discouraged in release versions that other modules may depend on. replace directives are only applied in the main module; they're ignored by go install and go get in another module or outside modules. This leads to subtle differences in selected versions depending on what command was issued where. We felt that reporting an error was preferable to that ambiguity.

When was it decided that replace directives are discouraged from release versions? I've scanned through the vgo blog series and the current Go Modules reference and I can't find anything suggesting this.

This is also discussed in the design doc and at length in the previous discussion in #30515.

It seems to me that the design doc and the previous discussion is simply arguing consistency with pathological behaviors of go get when used to build executables, but I think the point of adding a global install command would be to fix those problems not to further entrench them.

Bryan C. Mills · Answer 16 · Thu Sep 24 2020 23:44:55 GMT+0800 (China Standard Time)

@aarzilli, if you want users to install using git clone && go build, you can just tell them to do that — adding go install pkg@version doesn't remove or diminish that option; it just also doesn't promote that option by giving it a more concise shorthand.

Patrick Baxter · Answer 17 · Thu Sep 24 2020 23:46:03 GMT+0800 (China Standard Time)

@aarzilli Just want to reiterate because I share much of your perspective: Currently go get will ignore replace directives. With this change it won't do that anymore (since it will only be for dep management). Now, go install instead will error on replace directives indicating that you should be git cloning this to build it. So it will refuse to build the global binary with an alternative dep graph, which is good!

Bryan C. Mills · Answer 18 · Thu Sep 24 2020 23:50:53 GMT+0800 (China Standard Time)

When was it decided that replace directives are discouraged from release versions? I've scanned through the vgo blog series and the current Go Modules reference and I can't find anything suggesting this.

See https://research.swtch.com/vgo-principles#cooperation:

[W]hat we want to avoid, is an ecosystem that is fragmented, split into groups of packages that can’t work with each other.

replace directives inherently fragment the ecosystem. They indicate: “use this variant of the selected module instead of the version that everyone else sees”, and replacements are often incompatible with each other. They are a fine tool for experimentation, and in some cases a reasonable short-term option to work around a problem while you negotiate a fix that works for everyone, but they are not in line with the principle of cooperation.

Alessandro Arzilli · Answer 19 · Fri Sep 25 2020 00:04:22 GMT+0800 (China Standard Time)

@mvdan I don't disagree that it would be nice to use goproxy for this, but can it be? Judging from the list of caveats that accompany this design my impression is that it can't, in its current state.

@peebs Agreed, making go get error on replace directives is a good thing. I just don't like anything else. If the proposal becomes "go get errors when it encounters a replace directive" I'm 100% behind it.

@bcmills If go install becomes the default community meme for installing tools it will be antisocial not to use it. Besides "just don't use it if you don't like it" is a fully general answer to any proposal, if that was the rule of the land almost all proposals could be accepted. Then we could be like C++: everything's there just use the parts you like.

replace directives inherently fragment the ecosystem

I disagree. They do no such thing on an executable, you can have as many incompatible replace directives as you want on different executables without any interoperability problems. And if this is the belief of the go team then the documentation of replace should be changed to reflect that.

Tv · Answer 20 · Fri Sep 25 2020 01:51:47 GMT+0800 (China Standard Time)

@mvdan #40276 (comment)

Using the VCS repository directly is:

slower, since you have to download a lot more (module zips are generally much smaller)

You can use git clone --depth=1 if you don't want the VCS history.

You can also download a zip or tar from github, if you think zips are somehow inherently better than git.

Daniel Martí · Answer 21 · Fri Sep 25 2020 01:54:39 GMT+0800 (China Standard Time)

It's not just about the history. A repository can contain many modules, which is somewhat common. And module zips strip directories and files which aren't part of the set of Go packages.

Daniel Martí · Answer 22 · Fri Sep 25 2020 01:58:52 GMT+0800 (China Standard Time)

@aarzilli pretty much all the major points you bring up are already covered in the current proposal, mainly in the "Rationale" and "FAQ" sections. If you simply disagree with the conclusions, I would almost suggest that you should write a competing proposal, because you are essentially suggesting something entirely different.

We all agree that this problem needs solving, so I don't see a point in stopping or declining this proposal (which has had almost entirely positive feedback) unless there is an equally well thought-out competing proposal that should be considered instead.

Jay Conrod · Answer 23 · Fri Sep 25 2020 02:00:05 GMT+0800 (China Standard Time)

I disagree. They do no such thing on an executable, you can have as many incompatible replace directives as you want on different executables without any interoperability problems. And if this is the belief of the go team then the documentation of replace should be changed to reflect that.

Part of the rationale of this proposal is that you should be able to install an executable either within a dependent module (applying the constraints and replace directives from that module) or globally (without a main module). The former case is important for tools like code generators or analyzers that might be needed at specific versions or with modifications.

If we honor replace directives in the latter case, module authors will rely more heavily on replace directives, making the former case non-viable. I don't think that's an acceptable tradeoff when we can serve both cases.

We also can't honor directory replace directives at all. When something is downloaded into the module cache, relative and absolute systems from another repository aren't meaningful. Honoring some replace directives but not others adds too much complication.

One more point about downloading from VCS directly: if we completely step outside the module system for downloading code (which is what's being suggested), not only is it much slower (typically 5-7x, even with --depth=1), you're also throwing out safety: there's no go.sum file, so without the checksum database, you have no guarantee that the repository sends you the same bits as anyone else.

Alessandro Arzilli · Answer 24 · Fri Sep 25 2020 02:32:05 GMT+0800 (China Standard Time)

@mvdan I believe to have refuted the points covered in the proposals, if you disagree with my rebuttal I would ask you to point the error in my way rather than simply stating that I am wrong.

If you simply disagree with the conclusions, I would almost suggest that you should write a competing proposal, because you are essentially suggesting something entirely different.

To my knowledge this standard has never been applied to a proposal. For example the try proposal was rejected without a competing proposal being accepted.

@jayconrod

Part of the rationale of this proposal is that you should be able to install an executable either within a dependent module (applying the constraints and replace directives from that module) or globally (without a main module). The former case is important for tools like code generators or analyzers that might be needed at specific versions or with modifications.

I think this is a bad idea that doesn't solve any problem and introduce new ones of its own. First off it is very strange, almost antithetical, that a global install command should have a behavior that depends on the current directory. Secondly, let's make an example. Let's say I'm building a module that needs a code generator, let's call it goblah. I will have this directive in one of my source files:

//go:generate goblah -some -options

Then I add to go.mod the following:

require github.com/blahworld/goblah v31.0.0

Thinking this is enough to keep track of the goblah tool dependency. But if I simply tell contributors to run go generate this won't work, go generate does not automatically install the dependency and if a version of goblah happens to be on PATH, that's what will be used, regardless of its version or how it was built. People could even end up using odious version 29.0.3.

Soon enough I'll have to write gen.sh script like this:

#!/bin/bash
go-global-install github.com/blahworld/goblah && go generate

(I'm using go-general-install here as a placeholder for whatever the name of a global install will be). This has two problems:

It isn't much better than writing the version inside gen.sh directly
it's going to override the user's choice of what version is globally installed, maybe the user really needed version 29.0.3

What's really needed here is a command that will download an external tool, build it as specified in the go.mod file and execute it. Let's call this command go mod execute-ext-tool, then the go:generate directive could be changed to:

//go:generate go mod execute-ext-tool goblah -some -options

This way you could ensure that your builds are never soiled by goblah on versions other than 31.0.0 while your user can keep living on the 29.0.3 sin.

One more point about downloading from VCS directly: if we completely step outside the module system for downloading code (which is what's being suggested),

That's not what I'm suggesting. I'm suggesting that whatever is implemente should behave as if git clone && go build almost 100% of the time. For example go-global-install could request a special version to goproxy, in this format: github.com/blahworld/goblah@main:v31.0.0, then goproxy would know to prepare and serve an archive of the working directory retrieved from the repository at version 31.0.0. This archive could be checksummed in a similar way and wouldn't conflict with goblah@v31.0.0 due to being in a different namespace.

But the implementation details aren't as important as the principle that it should behave as if git clone && go build almost 100% of the time.

thepudds · Answer 25 · Fri Sep 25 2020 02:43:09 GMT+0800 (China Standard Time)

FWIW, my personal take is I think non-filepath replace directives should be supported for go install cmd/foo@v1.2.3.

A decent percentage of large & frequently used open source Go projects have replace directives in their go.mod files. There are different reasons, but open source is fundamentally messy, and it takes time to upstream fixes or otherwise resolve issues.

If go install cmd/foo@v1.2.3 cannot use non-filepath replace, it is a hint that go install / go get does not scale to large projects, which seems at odds with the overall philosophy of the Go project.

During the replace discussion, I think some people suggested there is a tension between:

People who want to use a tool as published by the tool author (e.g., including replace directives).
vs.
People who want to track their dependencies.

During the discussion, some people I think have suggested that not supporting replace in go install cmd/foo@v1.2.3 under the current proposal makes it easier for consumers to track tool dependencies.

That ends up not really being true if a higher percentage of projects resolve this limitation by choosing to still use replace but pick another distribution mode (publishing binaries, install script, git commands, or similar). Those alternative distribution modes don't play well with a tools.go approach.

One can certainly debate what will happen more frequently in terms of how the proposed "no replace" limitation gets resolved:
(a) in a way that does plays well with tools.go approach, vs.
(b) in a way that does not play well with a tools.go approach (e.g., publishing a binary or similar).

Personally, my guess is the "no replace" limitation pushes more of the largest and most often consumed projects towards (b) at a higher frequency than (a). At a minimum, some publishers choosing (b) eats into the benefit delivered by (a).

Also, whenever someone chooses a separate distribution mode (binaries, install script, git, ...), that also has other negative ecosystem impacts, including they are side-stepping all the "securing the Go ecosystem" work of the checksum DB, and that happens to be occurring at almost the worst point in the delivery chain -- at the point of consumption by the ultimate end user.

Patrick Baxter · Answer 26 · Fri Sep 25 2020 02:49:24 GMT+0800 (China Standard Time)

@peebs Agreed, making go get error on replace directives is a good thing. I just don't like anything else. If the proposal becomes "go get errors when it encounters a replace directive" I'm 100% behind it.

@aarzilli Just to absolutely make sure we're on the same page (apologies if this is already clear): This is the current proposal (after much debate, me included) except that its called go install. The original proposal for go get -b would have ignored replace, but now go install will error.

It also potentially opens the door up down the road to respect some replace directives without any breaking changes. Since some of the errors would now just do the right thing.

Alessandro Arzilli · Answer 27 · Fri Sep 25 2020 03:13:08 GMT+0800 (China Standard Time)

@peebs The proposal is:

We propose augmenting the go install command to build and install packages
at specific versions, regardless of the current module context.

go install golang.org/x/tools/gopls@v0.4.4

To eliminate redundancy and confusion, we also propose deprecating and removing
go get functionality for building and installing packages.

I'm objecting to the way the first part (go install change) is designed. I'm fine with the second part and I'm also fine with changing go install to be a global install command, as long as it's behavior is approximately the same as git clone && go build.

Jay Conrod · Answer 28 · Fri Sep 25 2020 03:13:43 GMT+0800 (China Standard Time)

@aarzilli

I believe to have refuted the points covered in the proposals, if you disagree with my rebuttal I would ask you to point the error in my way rather than simply stating that I am wrong.

I've referred you back to the design doc in a couple places where your points were covered already.

Other points about expected behavior I think are more of a matter of opinion. We've covered a lot of opinions, and this proposal is a compromise.

I think this is a bad idea that doesn't solve any problem and introduce new ones of its own.

You're not winning me over with this.

First off it is very strange, almost antithetical, that a global install command should have a behavior that depends on the current directory.

It doesn't depend on the current directory. That's the point. If you want to install an executable without modifying the go.mod file in the current directory or changing to another directory, you don't have to change to another directory anymore.

Secondly, let's make an example. Let's say I'm building a module that needs a code generator, let's call it goblah. ...

You wouldn't use the new functionality in this situation. You would use it to install tools that aren't specifically related to any project like gopls or goimports.

Your example mostly works today, and we're trying to keep it working by not encouraging widespread use of replace.

You can use go run to avoid installing an executable directly:

//go:generate go run example.com/tool

Or in a script with non-global go install:

#!/bin/bash

GOBIN=bin go install example.com/tool1 example.com/tool2 # versions come from our go.mod
bin/tool1
bin/tool2

Or with go build:

#!/bin/bash

go build -o bin/ example.com/tool1 example.com/tool2 # versions come from our go.mod
bin/tool1
bin/tool2

These commands work consistently. They use go.mod in the current module to select versions.

What's really needed here is a command that will download an external tool, build it as specified in the go.mod file and execute it. Let's call this command go mod execute-ext-tool, then the go:generate directive could be changed to:

You're describing go run. It works today, though it's a little slow because we don't cache the linked binary. We can improve that.

This proposal is not meant to address that use case.

That's not what I'm suggesting. I'm suggesting that whatever is implemente should behave as if git clone && go build almost 100% of the time. For example go-global-install could request a special version to goproxy, in this format: github.com/blahworld/goblah@main:v31.0.0, then goproxy would know to prepare and serve an archive of the working directory retrieved from the repository at version 31.0.0. This archive could be checksummed in a similar way and wouldn't conflict with goblah@v31.0.0 due to being in a different namespace.

The important difference is that a module zip file contains files from one module, not from other modules in the same repository. Directory replace directives always point outside the current module, most commonly to other modules in the same repository, but sometimes to completely different repositories. That's why we can't honor them.

Even if we packed all modules into a single zip file as you're suggesting, we couldn't make directory replacements work all the time.

Alessandro Arzilli · Answer 29 · Fri Sep 25 2020 03:22:43 GMT+0800 (China Standard Time)

@jayconrod

It doesn't depend on the current directory. That's the point. If you want to install an executable without modifying the go.mod file in the current directory or changing to another directory, you don't have to change to another directory anymore.

But it does. It's going to look at the go.mod file that's either in the current directory or in a parent directory and look up versions and replaces there.

You wouldn't use the new functionality in this situation. You would use it to install tools that aren't specifically related to any project like gopls or goimports.

Ok, so why is this even mentioned in the proposal? Clearly tracking tool dependencies shouldn't influence the design of a global install command.

The important difference is that a module zip file contains files from one module, not from other modules in the same repository.

Yes. Current module zips are like this. But this doesn't work for a global install command, so a different type of archive needs to be added.

Directory replace directives always point outside the current module, most commonly to other modules in the same repository, but sometimes to completely different repositories. That's why we can't honor them.

Why? Go get has the ability to recursively download dependencies already. You download the @main:version archive, unpack it and run go build inside it. I'm not really sure what's impossible about this.

Even if we packed all modules into a single zip file as you're suggesting, we couldn't make directory replacements work all the time.

Under what circumstances would they not work? Clearly .. replacements that point outside the repository won't work but nobody will expect that to work and that matches the behavior of git clone && go build anyway.

Jay Conrod · Answer 30 · Fri Sep 25 2020 03:43:34 GMT+0800 (China Standard Time)

@aarzilli

But it does. It's going to look at the go.mod file that's either in the current directory or in a parent directory and look up versions and replaces there.

No. The proposal says:

go install runs in module mode, regardless of whether a go.mod file is
present. If GO111MODULE=off, go install reports an error, similar to
what go mod download and other module commands do.
go install acts as if no go.mod file is present in the current directory
or parent directory.
No module will be considered the "main" module.

Ok, so why is this even mentioned in the proposal? Clearly tracking tool dependencies shouldn't influence the design of a global install command.

It's mentioned because it's a design constraint. In introducing a new feature, we don't want to change the ecosystem in a way that breaks existing usage.

The section "Why can't module replace directives be used?" discusses this: it includes an example similar to the one you gave earlier.

Yes. Current module zips are like this. But this doesn't work for a global install command, so a different type of archive needs to be added.
...
Under what circumstances would [directory replacements] not work?

Any directory replacement outside the current repository. Any absolute path replacement.

Even if we could honor all replacements though, I don't think we should for the same reasons we're not honoring module replacements: if used widely, they prevent other modules from depending on modules that provide tools.

Alessandro Arzilli · Answer 31 · Fri Sep 25 2020 04:05:34 GMT+0800 (China Standard Time)

@aarzilli

But it does. It's going to look at the go.mod file that's either in the current directory or in a parent directory and look up versions and replaces there.

No. The proposal says:
* go install runs in module mode, regardless of whether a go.mod file is
  present. If GO111MODULE=off, go install reports an error, similar to
  what go mod download and other module commands do.

* go install acts as if no go.mod file is present in the current directory
  or parent directory.

* No module will be considered the "main" module.

Then I don't understand the paragraphs following the words "However, there are two reasons to avoid applying replace directives at all", it seems to me that they mean nothing.

For example:

applying replace directives would create inconsistency for users inside
and outside a module.

It wouldn't because it would behave the same, right?

When a package is built within a module with go build or
go install, only replace directives from the main module are applied, not
the module providing the package.

What else would you be able to install besides a main package?

When a package is built outside a module with
go get, no replace directives are applied.

Why is the current behavior of go get relevant when the impetus behind this change is that it is wrong?

If go install applied replace
directives from the module providing the package, it would not be consistent
with the current behavior of any other build command.

What about the main build command, go build?

Second, if go install applied replace directives, it would take power away
from developers that depend on modules that provide tools.

If their go.mod file is not considered how would this take away power from them?

For example, suppose
the author of a popular code generation tool gogen forks a dependency
genutil to add a feature. They add a replace directive pointing to their
fork of genutil while waiting for a PR to merge.

Where would they add a replace directive if the go.mod isn't considered?

Ok, so why is this even mentioned in the proposal? Clearly tracking tool dependencies shouldn't influence the design of a global install command.

It's mentioned because it's a design constraint. In introducing a new feature, we don't want to change the ecosystem in a way that breaks existing usage.

But you are breaking 10% of the ecosystem with this?

The section "Why can't module replace directives be used?" discusses this: it includes an example similar to the one you gave earlier.

My impression is that the intent was to obey the go.mod file in the current (or parent) directory. If that's not the case I don't think that section actually makes a case against applying the replace directives, if the replace directives of the current go.mod can't be applied then the replace directives of the tool being installed can be applied.

Yes. Current module zips are like this. But this doesn't work for a global install command, so a different type of archive needs to be added.
...
Under what circumstances would [directory replacements] not work?

Any directory replacement outside the current repository. Any absolute path replacement.

Nobody would do that and expect it to work. It also wouldn't work with git clone && go build which is what the global install command should match.

Even if we could honor all replacements though, I don't think we should for the same reasons we're not honoring module replacements: if used widely, they prevent other modules from depending on modules that provide tools.

I think the usefulness of replace directives vastly outstrips the usefulness of using an internal package of a tool as a dependency. But if you think this way I think you should propose a change to the documentation to say that replace directives shouldn't appear in published mod files ever. We could see what the community thinks about that rather than burying this change in the implementation of go install.

Jay Conrod · Answer 32 · Fri Sep 25 2020 05:02:30 GMT+0800 (China Standard Time)

@aarzilli

It wouldn't because it would behave the same, right?

No. When you run go install example.com/A/pkg from the module example.com/Main, it applies require and replace directives from example.com/Main, not example.com/A.

When you run go get example.com/A/pkg in module mode from outside any module, it silently ignores replace directives.

Many people expect that replace directives are applied in either or both cases, but there's no command that works that way.

When you run go install example.com/A/pkg@v1.0.0 (this proposal), it will report an error if the module example.com/A has replace directives. That way, there's no ambiguity.

What else would you be able to install besides a main package?

The main module is the module defined by the go.mod file in the current directory or a parent directory. It's likely different than the module providing the main package in this context.

Why is the current behavior of go get relevant when the impetus behind this change is that it is wrong?

go get is how executables are installed in module mode today, so that's the relevant thing to compare against. It's not wrong, just inconvenient and overloaded. If you want to install an executable globally in module mode, ignoring the module in the current directory, you have to do something like this:

$ (cd /; GO111MODULE=on go get example.com/A/pkg)

What about the main build command, go build?

go build applies replace directives from the module in the current directory, not from anywhere else. Same as go test, go run, go install, go list.

If their go.mod file is not considered how would this take away power from them?

If go install example.com/A/pkg@v1.0.0 applied replace directives from example.com/A, then the author of example.com/A might release a version that didn't build without replace directives.

If the module example.com/B depends on example.com/A, they are now broken unless they copy those replace directives, which may not even be feasible: the replacements from example.com/A might conflict with replacements from example.com/B.

By encouraging replace beyond development within one module, we take power away from module users and give it to module authors.

But you are breaking 10% of the ecosystem with this?

Who does this break? This doesn't change the interpretation of existing go install commands or any other command. It addresses a use case that wasn't adequately covered before.

If a module doesn't build without replace, then there's no build command that works outside that module.

If a module has replace but it can safely be ignored, then commands that work today will keep working (though we may deprecate go get for that purpose).

But why include replace at release versions if it's not needed? It only causes confusion.

I surveyed modules with main packages in "Appendix: usage of replace directives". I think the "soft fork" case is the only one we could realistically address, and that's 4% of modules.

My impression is that the intent was to obey the go.mod file in the current (or parent) directory. If that's not the case I don't think that section actually makes a case against applying the replace directives, if the replace directives of the current go.mod can't be applied then the replace directives of the tool being installed can be applied.

Covered above.

Nobody would do that and expect it to work. It also wouldn't work with git clone && go build which is what the global install command should match.

It sounds like the only advantage of this is that directory replace directives could be applied in modules that replace other modules in the same repository. That's less than 3% of modules with main packages. That's also a problem that's entirely within module authors' power to solve. They can tag releases for each module pointing to the same commit, then tag those commits.

But if you think this way I think you should propose a change to the documentation to say that replace directives shouldn't appear in published mod files ever. We could see what the community thinks about that rather than burying this change in the implementation of go install.

We're in the process of writing more module documentation. Usage of replace is one of topics we have planned and will likely advise caution around release versions.

Alessandro Arzilli · Answer 33 · Fri Sep 25 2020 13:47:26 GMT+0800 (China Standard Time)

@aarzilli

It wouldn't because it would behave the same, right?

No. When you run go install example.com/A/pkg from the module example.com/Main, it applies require and replace directives from example.com/Main, not example.com/A.

Just to be clear, this is its current behavior, not the new behavior being proposed here?

When you run go get example.com/A/pkg in module mode from outside any module, it silently ignores replace directives.

Many people expect that replace directives are applied in either or both cases, but there's no command that works that way.

When you run go install example.com/A/pkg@v1.0.0 (this proposal), it will report an error if the module example.com/A has replace directives. That way, there's no ambiguity.

But if you run go install example.com/A/pkg from inside example.com/A/pkg does it use require and replace directives from its own go.mod file? Because if it doesn't then go install behavior is inconsistent with go build and go test. I have a better solution go install example.com/A/pkg always applies require and replace directives from example.com/A/pkg, this way go install, go build and go test all behave the same way.

What else would you be able to install besides a main package?

The main module is the module defined by the go.mod file in the current directory or a parent directory. It's likely different than the module providing the main package in this context.

Now I'm confused again. Could you rewrite the proposal to make it clear where you are talking about current behavior of go install and where you are talking about proposed new behavior of go install?

Why is the current behavior of go get relevant when the impetus behind this change is that it is wrong?

go get is how executables are installed in module mode today, so that's the relevant thing to compare against. It's not wrong, just inconvenient and overloaded.

I think we both agree that it's wrong since you are proposing that the new thing should error when it encounters a replace directive.

What about the main build command, go build?

go build applies replace directives from the module in the current directory, not from anywhere else. Same as go test, go run, go install, go list.

What you are saying it is that go build example.com/some/package is going to apply replace/require directives from the current directory, even if the current directory has nothing to do with example/com/some/package. How often do you think this behavior is used? Do you think this is what users expect to happen? What if instead it always applied the replace/require directives of example.com/some/package?

If their go.mod file is not considered how would this take away power from them?

If go install example.com/A/pkg@v1.0.0 applied replace directives from example.com/A, then the author of example.com/A might release a version that didn't build without replace directives.

If the module example.com/B depends on example.com/A, they are now broken unless they copy those replace directives, which may not even be feasible: the replacements from example.com/A might conflict with replacements from example.com/B.

Ok, I get it now. Thank you. I think a better solution is to always apply replace directives from example.com/A, that way the author of example.com/A won't find out about things being broken from its users.

By encouraging replace beyond development within one module, we take power away from module users and give it to module authors.

Ok, but we are talking about tool dependencies here and we have already established that there are better ways to solve that problem that using a global install command to manage local dependencies. So this isn't a problem, the module user, in this case, can use go mod run-external-tool (or whatever it gets called) to build and run the external tool with requires and replaces from the current go.mod file applied.

I think that by erroring on replace directives you also take power away from module users (that have to wait for PR to be merged) and give it to module authors.

But you are breaking 10% of the ecosystem with this?

Who does this break? This doesn't change the interpretation of existing go install commands or any other command. It addresses a use case that wasn't adequately covered before.

If the new global install command becomes the community meme for installing programs then 10% of the existing modules are no longer installable.

If a module doesn't build without replace, then there's no build command that works outside that module.

How about we change this to "all build commands should work outside that module"?

Nobody would do that and expect it to work. It also wouldn't work with git clone && go build which is what the global install command should match.

It sounds like the only advantage of this is that directory replace directives could be applied in modules that replace other modules in the same repository.

No. I think all replace directives that work with git clone && go build should be respected.

Patrick Baxter · Answer 34 · Fri Sep 25 2020 14:49:40 GMT+0800 (China Standard Time)

@aarzilli I think making these arguments against this proposal is counter productive because its really a big win:

With this change we now have a versioned global install tool that can be trusted to work as expected. This didn't exist in module land or gopath land and now it does. Just use go install pkg@version. It won't work on every project, but that would never be true and many projects rely on make files, other binaries, and/or c libraries anyway. Erroring on replaces is a conservative and correct change for expectations here that leaves room for respecting some replaces down the road without breaking anything.

There is tension between the use cases of global (no local module context) installation of a binary and building an external binary in the context of a local module's dependencies (as @thepudds mentioned). I too think the latter case is so odd as to confuse the tooling, but, this change is a big win and gives a clear path forward for having a distinguished global install command. Its also a big clean up in regards to go get being overloaded since modules.

I think either asking go install pkg@version to use replaces or asking to eliminate the 2nd use case entirely would have to be separate proposals, but stopping this one is a step backward. Its a great advancement/compromise and its provides functionally many have wanted for a very long time while cleaning up the tooling ux.

Daniel Martí · Answer 35 · Fri Sep 25 2020 16:40:56 GMT+0800 (China Standard Time)

First of all, thank you for sharing your concerns, @aarzilli. It shows that you're passionate about the issue at hand, like the rest of us. Having said that, the way you've come to this thread to suddenly push back against the idea comes as a surprise, at least.

I believe to have refuted the points covered in the proposals, if you disagree with my rebuttal I would ask you to point the error in my way rather than simply stating that I am wrong.

Jay is already patiently clarifying your doubts about the proposal, so I won't step in the middle of the conversation.

To my knowledge this standard has never been applied to a proposal. For example the try proposal was rejected without a competing proposal being accepted.

The bar for big language changes is almost impossibly high. There haven't been any big changes since Go 1.0, and we can't even agree if error handling is a problem worth solving. In comparison, cmd/go evolves rapidly (Modules being the prime example), and everyone seems to agree that the current state of "download a program" is broken. So, yes, I still think that the only reasonable way to push back on this proposal is to, well, propose something better.

Please bear in mind that this problem has been in our minds for years; see #30515 and the even older issues before that. We've also discussed and iterated on this particular proposal for nearly a year in our monthly golang-tools calls. Yes, the current proposal has significant tradeoffs, but like @peebs very well put it, the solution is a step in the right direction and leaves the door open for eventually supporting replace directives in the future.

No. I think all replace directives that work with git clone && go build should be respected.

The proposal tackles this issue head-on, so I'm not sure what else to say on the topic. To be perfectly clear, it seems to me like you have multiple options if you absolutely want replace directives to always be obeyed:

Support this proposal, and once implemented, propose a way to extend the design to support replace directives (since the door is left open)
Make a competing proposal with a better design, to be accepted instead of this one
Insist that we reject this proposal with no other plan in mind

I personally think option 1 is the most sensible; smaller incremental steps, and the first step is already implemented and ready to ship in 1.16 if this proposal is accepted. Option 2 could also be a good option if you actually come up with a better design, but with the big caveat that such a better solution would take at least one or two more cycles to be available to users.

Perhaps I'm misunderstanding you, but you seem to be going for option 3, which would be unfortunate as we would simply not move forward in any direction anytime soon.

thepudds · Answer 36 · Fri Sep 25 2020 17:47:43 GMT+0800 (China Standard Time)

making these arguments against this proposal is counter productive because its really a big win

I think proposals frequently get narrowed or broadened as part of the proposal discussion process (e.g., random recent example is @jayconrod comment in #40364 (comment)).

Personally, I am 👍 on the behavior as currently implemented in tip, but 👎 on the proposal itself as written.

At the conclusion of #30515 and some contemporaneous discussion in golang-tools calls, my (perhaps incorrect) understanding at the time was that the question of whether or not to support non-directory replace directives was being postponed and being placed outside the scope of this new proposal, and that erroring out was a more conservative approach that allows for future change.

As written, the proposal seems to too strongly indicate that non-directory replace directives won't be supported, at least as I read it, and I worry people will refer back to this proposal (if accepted as written) as deciding the fate of non-directory replace directives.

I would be 👍 on the proposal if it was clearer on the point that this proposal is not deciding the fate of non-directory replace directives, and that it provides reasonable behavior (an error) that keeps more options open for the future.

One way to do that would be expanding this paragraph slightly:

If this behavior turns out not to be suitable (for example, authors prefer to
keep replace directives in go.mod at release versions and understand that
they won't affect users), then we could start ignoring replace directives in
the future, matching current go get behavior.

Finally, I understand there are thorny issues here and competing concerns, and I certainly appreciate the effort that has already gone into creating, discussing, and implementing this proposal. ❤️

Daniel Martí · Answer 37 · Fri Sep 25 2020 18:00:09 GMT+0800 (China Standard Time)

I would be +1 on the proposal if it was clearer on that point.

Aside from it must not contain `replace` or `exclude` and Why can't module replace directives be used?, what do you think could make the proposal clearer?

I worry people will refer back to this proposal (if accepted as written) as deciding the fate of non-directory replace directives.

I hope that won't be the case, as long as we make it clear that we are leaving the door open for supporting replace directives in the future.

Alessandro Arzilli · Answer 38 · Fri Sep 25 2020 20:29:47 GMT+0800 (China Standard Time)

@mvdan

First of all, thank you for sharing your concerns, @aarzilli. It shows that you're passionate about the issue at hand, like the rest of us. Having said that, the way you've come to this thread to suddenly push back against the idea comes as a surprise, at least.

The old discussion was abruptly closed, I was busy at the time and I sort of lost track of where the discussion went after that.

Yes, the current proposal has significant tradeoffs, but like @peebs very well put it, the solution is a step in the right direction and leaves the door open for eventually supporting replace directives in the future.

I don't see it this way. The proposal makes a strong (if invalid) argument against supporting replaces.

So, yes, I still think that the only reasonable way to push back on this proposal is to, well, propose something better.

I think at this point I have, throughout this discussion. But the merits of this proposal should be discussed in this proposal.

Daniel Martí · Answer 39 · Fri Sep 25 2020 21:17:56 GMT+0800 (China Standard Time)

The old discussion was abruptly closed, I was busy at the time and I sort of lost track of where the discussion went after that.

I disagree on both accounts; @jayconrod mentioned that him and myself started working on this proposal a month before closing the issue, and it did get closed since noone seemed to disagree with moving the discussion to a proposal. We also linked this issue explicitly.

I don't see it this way.

That appears to be more of an optics problem, then. This proposal is not "let's actively deprecate the use of replace directives in tagged releases", and we went to great lengths to make sure that they could be supported in the future. This feedback seems to align with what @thepudds says - do you have any suggestion other than "reject this proposal"?

I think at this point I have, throughout this discussion.

Right, and I think minor adjustments to the proposal are more than reasonable. But if your idea is based on an entirely different premise (that this can't work without supporting replace directives from day one), then I do think it's essentially a different proposal, even if it solves the same core problem.

Alessandro Arzilli · Answer 40 · Fri Sep 25 2020 22:20:34 GMT+0800 (China Standard Time)

The old discussion was abruptly closed, I was busy at the time and I sort of lost track of where the discussion went after that.

I disagree on both accounts;

You disagre on both accounts? As in: you disagree that I was busy?

@jayconrod mentioned that him and myself started working on this proposal a month before closing the issue, and it did get closed since noone seemed to disagree with moving the discussion to a proposal. We also linked this issue explicitly.

I don't dispute any of this nor do I have any problem with it, but I don't see why this is important: random people comment on proposals all the time.

I don't see it this way.

That appears to be more of an optics problem, then. This proposal is not "let's actively deprecate the use of replace directives in tagged releases", and we went to great lengths to make sure that they could be supported in the future. This feedback seems to align with what @thepudds says - do you have any suggestion other than "reject this proposal"?

I think in my discussion with @jayconrod I have provided some constructive feedback on how to improve this proposal by making the global install command respect replace directives. Let's say we take this proposal, remove everything that mentions the replace directive and instead say "the global install command is going to respect module replace directives, directory replaces can not be supported at the moment due to a limitation of goproxy which will be addressed in a future release" and explain how it plans to support directory replaces. I'd be ok with that.

I think at this point I have, throughout this discussion.

Right, and I think minor adjustments to the proposal are more than reasonable. But if your idea is based on an entirely different premise (that this can't work without supporting replace directives from day one), then I do think it's essentially a different proposal, even if it solves the same core problem.

Let's say I make a competing proposal, the first thing people would write as a comment is "didn't you read #40276? We can't and shouldn't support replace directives" and then we'd be back here.

Daniel Martí · Answer 41 · Fri Sep 25 2020 22:35:30 GMT+0800 (China Standard Time)

You disagre on both accounts? As in: you disagree that I was busy?

I disagree that the proposal was abruptly closed, or that it wasn't clear where the discussion was moving.

random people comment on proposals all the time.

I also left a reply just before the issue was closed. I was the author of that issue, so I'm not sure if I qualify as a random person :)

explain how it plans to support directory replaces

This would be a major change to how the module proxy protocol works, and would affect a lot of other pieces of software including the existing proxies and checksum databases, as well as pkg.go.dev and any other tool interacting with the proxy protocol. I definitely don't think we should be making any promises until a proposal for such a redesign is made and accepted.

"didn't you read #40276? We can't and shouldn't support replace directives"

This would be the main difference with your proposal - you would start from a completely different premise around replace directives. As long as you make that clear, I doubt anyone would close your proposal as a duplicate.

Jay Conrod · Answer 42 · Sat Sep 26 2020 02:50:03 GMT+0800 (China Standard Time)

@aarzilli

Just to be clear, this is its current behavior, not the new behavior being proposed here?

Yes. Currently, all module commands honor replace and exclude directives in the main module's go.mod file, that is, the go.mod file in the current directory (or any parent). replace and exclude directives in all other go.mod files are ignored.

But if you run go install example.com/A/pkg from inside example.com/A/pkg does it use require and replace directives from its own go.mod file?

Yes. If example.com/A is the main module, then its replace directives apply.

I have a better solution go install example.com/A/pkg always applies require and replace directives from example.com/A/pkg, this way go install, go build and go test all behave the same way.

go build and go test don't work that way. No module command does.

This would completely change the interpretation of replace, and it would break a lot of modules.

Could you rewrite the proposal to make it clear where you are talking about current behavior of go install and where you are talking about proposed new behavior of go install?

The new behavior is specified carefully in the "Proposal" section. The current behavior is explained in "Current go install and go get functionality".

You may also want to read the reference documentation for more clarity on the current behavior:

I think we both agree that it's wrong since you are proposing that the new thing should error when it encounters a replace directive.

I don't agree.

My personal opinion is that go install pkg@version should ignore all replace directives. That would make it behave like go get does when invoked in module mode outside a module. That would let module authors leave replace directives in place in release versions, knowing that they wouldn't affect users.

As I mentioned earlier though, this proposal was a compromise, and we decided to go with something very strict. If we find that's not the right behavior, we can relax it a bit, either by ignoring replace directives or applying module replace directives only.

What you are saying it is that go build example.com/some/package is going to apply replace/require directives from the current directory, even if the current directory has nothing to do with example/com/some/package. How often do you think this behavior is used? Do you think this is what users expect to happen? What if instead it always applied the replace/require directives of example.com/some/package?

No change to go build is proposed here. This is already the current behavior, and it's used all the time.

I certainly hope people expect this. If I run go build example.com/A/pkg from my main module example.com/Main, and the main module replaces example.com/A, I'd expect to build the package from my replacement, not something else.

Ok, but we are talking about tool dependencies here and we have already established that there are better ways to solve that problem that using a global install command to manage local dependencies. So this isn't a problem, the module user, in this case, can use go mod run-external-tool (or whatever it gets called) to build and run the external tool with requires and replaces from the current go.mod file applied.

They can use go run, go build, or go install as they work today. No change is being proposed for this use case, and I don't see any reason to break it.

We are talking about it here because we're specifically trying to avoid breaking it.

I think that by erroring on replace directives you also take power away from module users (that have to wait for PR to be merged) and give it to module authors.

If a user needs to fork and send PRs for one or more modules, then this proposal is not for them. They can run go install (without a version) from their fork.

If you mean they need to wait for the author's PRs to be merged, they already have a problem with go get and all other commands, which ignore replace directives outside the main module.

The difference here is that go install pkg@version would report an error for replace instead of silently ignoring it.

If the new global install command becomes the community meme for installing programs then 10% of the existing modules are no longer installable.

Most (nearly all?) of these issues can be resolved by adjusting go.mod and tagging a new release.

We'll monitor this over time. If we need to change course, there are a couple ways we can do that, but as I've mentioned, we're starting with something strict so that those options are available to us without making a breaking change in the future.

I think in my discussion with @jayconrod I have provided some constructive feedback on how to improve this proposal by making the global install command respect replace directives. Let's say we take this proposal, remove everything that mentions the replace directive and instead say "the global install command is going to respect module replace directives, directory replaces can not be supported at the moment due to a limitation of goproxy which will be addressed in a future release" and explain how it plans to support directory replaces. I'd be ok with that.

You've suggested a large change to the current interpretation of replace directives, which would break lots of existing modules. You've also suggested a significant change to the proxy protocol, which would be expensive to implement across the ecosystem. These are significant drawbacks, and the benefits aren't at all clear to me. It would be difficult for such a proposal to be approved.

Jay Conrod · Answer 43 · Sat Sep 26 2020 02:50:56 GMT+0800 (China Standard Time)

@thepudds

I'm open to editing the proposal to discuss future modifications. The two possible paths we could take are 1) ignoring replace directives, or 2) applying only module replace directives, still reporting an error if a directory replace directive is present.

It would be good to talk about what kind of data would convince us to take one path or the other.

For (1), we should see how much of a hassle it is for authors to edit go.mod for release versions.

For (2) applying module replace directives, I'd like to be sure that modules that depend on tool-providing modules won't be seriously impacted by additional replace directives. I think we need a better story for tool dependencies in general (for example, an enhanced go run pkg@version or a new kind of require-like directive for tools. That won't happen for 1.16, so I'd like to defer making a decision on that for now.

Anything else on your mind in this area?

Jay Conrod · Answer 44 · Sat Sep 26 2020 02:52:49 GMT+0800 (China Standard Time)

One more comment for everyone: this feature is implemented at tip in CL 254365. You can try it out with golang.org/dl/gotip. Please experiment and let us know how it works for you.

There's a little more than a month before the 1.16 freeze (November) to make significant changes, and we'll have about a month after that before 1.16beta1 (December) to find and fix bugs.

Daniel Martí · Answer 45 · Sat Sep 26 2020 02:59:21 GMT+0800 (China Standard Time)

It would be good to talk about what kind of data would convince us to take one path or the other.

If 1.16 ships with what's on tip at the moment, perhaps we could run a survey a couple of months after launch to get user input. If we ask the right questions, I think that would probably be the best way to get actionable feedback data.

thepudds · Answer 46 · Sat Sep 26 2020 05:13:07 GMT+0800 (China Standard Time)

Hi @jayconrod, sorry to throw out another topic, but on tip, go install foo@v1.2.3 seems to accept a -modfile argument, but might not do anything with it? That might be a minor bug, but I was thinking about possible -modfile behavior with go install when this proposal was first posted, and curious what you or others might be thinking regarding -modfile...

For now, it could be reasonable to error out or otherwise disallow -modfile, but probably worth at least some brief thinking around what candidate future behavior might be just to see if there are any implications for the current proposal. Some sample questions about possible future behavior:

Example 1:

If you do:
go install -modfile=install-gogen.mod example.com/cmd/gogen@v1.2.3

Then immediately do the following (without a version):
go install -modfile=install-gogen.mod example.com/cmd/gogen

Do you end up with v1.2.3? (In other words, would require example.com/cmd/gogen v1.2.3 have been written down in install-gogen.md, and would it have been applied in the second invocation)?

Example 2:

If you do:
go install -modfile=install-gogen.mod example.com/cmd/gogen@v1.2.3

Then edit the install-gogen.mod file to adjust versions via require or add a replace for more control of dependencies used by gogen, and then do:
go install -modfile=install-gogen.mod example.com/cmd/gogen

Would it use the adjusted versions you specified in install-gogen.mod?

Example 3:

-u is not currently supported for go install (though I'm not sure I saw the rationale written down for why). If it was to be supported eventually, if you then did:
go install -u=patch -modfile=install-gogen.mod example.com/cmd/gogen@some-branch

And then immediately re-use the modfile:
go install -modfile=install-gogen.mod example.com/cmd/gogen@some-branch

Does it use the updated dependencies that were recorded in install-gogen.mod (and thereby providing more reproducibility for -u?)

In any event, could be reasonable to punt by just disallowing it for now, but curious to hear what you might think.

Bryan C. Mills · Answer 47 · Sat Sep 26 2020 05:21:46 GMT+0800 (China Standard Time)

@thepudds, we plan to reject -modfile; that part still needs to be implemented.
(See https://go-review.googlesource.com/c/go/+/254365/comment/11545721_87af54fe/.)

Alessandro Arzilli · Answer 48 · Sat Sep 26 2020 19:23:33 GMT+0800 (China Standard Time)

@mvdan

You disagre on both accounts? As in: you disagree that I was busy?

I disagree that the proposal was abruptly closed, or that it wasn't clear where the discussion was moving.

I said I lost track, not that it wasn't clear.

random people comment on proposals all the time.

I also left a reply just before the issue was closed. I was the author of that issue, so I'm not sure if I qualify as a random person :)

The random person in question would be me.

explain how it plans to support directory replaces

This would be a major change to how the module proxy protocol works, and would affect a lot of other pieces of software including the existing proxies and checksum databases, as well as pkg.go.dev and any other tool interacting with the proxy protocol. I definitely don't think we should be making any promises until a proposal for such a redesign is made and accepted.

Yes. This proposal would also be that proposal.

Alessandro Arzilli · Answer 49 · Sat Sep 26 2020 19:49:39 GMT+0800 (China Standard Time)

I have a better solution go install example.com/A/pkg always applies require and replace directives from example.com/A/pkg, this way go install, go build and go test all behave the same way.

go build and go test don't work that way. No module command does.

This would completely change the interpretation of replace, and it would break a lot of modules.

After thinking more about this, I think you are right that this isn't appropriate for most subcommands of cmd/go. When executing go X P where X is build, test, etc and P is a package there are two possibilities:

a) the directives in the current go.mod are applied
b) the directives in the module providing P are applied

Both options have merits. What doesn't ever have merit is "use the directives of the module providing P but error if there is a replace directive" and by virtue of the fact that a global install command should not have outcomes that depend on the directory it is run in, it's clear that for the global install command the right behavior is (b).

go get is how executables are installed in module mode today, so that's the relevant thing to compare against. It's not wrong, just inconvenient and overloaded.

I think we both agree that it's wrong since you are proposing that the new thing should error when it encounters a replace directive.

I don't agree.

Then why are you proposing a command with a different behavior and that the current behavior of go get should be deprecated and removed?

My personal opinion is that go install pkg@version should ignore all replace directives. That would make it behave like go get does when invoked in module mode outside a module. That would let module authors leave replace directives in place in release versions, knowing that they wouldn't affect users.

I don't think people add replace directives to go.mod expecting them to be ignored. I think the global install command should respect replace directives so that module authors can leave replace directives in place knowing that they will be respected.

They can use go run, go build, or go install as they work today. No change is being proposed for this use case, and I don't see any reason to break it.

We are talking about it here because we're specifically trying to avoid breaking it.

How would respecting replace directives break it?

I think that by erroring on replace directives you also take power away from module users (that have to wait for PR to be merged) and give it to module authors.

If a user needs to fork and send PRs for one or more modules, then this proposal is not for them. They can run go install (without a version) from their fork.

I'm the author of a tool, I send a PR to one of my dependencies, unfortunately the author has taken a long vacation to Mars and the PR will be merged 6 months from now. In the meantime I fork the dependency and add a temporary replace directive. People try to install my tool using go install but it doesn't work, because replace directives aren't respected. How am I not a module user, and how isn't this taking power away from me and giving it to the Mars vacationer?

If you mean they need to wait for the author's PRs to be merged, they already have a problem with go get and all other commands, which ignore replace directives outside the main module.

Yes, they are going to have a problem with go get. This is a bug. It should be fixed. We're talking about a tool, something that builds to an executable, I'm not sure what other commands you think are relevant.

I think in my discussion with @jayconrod I have provided some constructive feedback on how to improve this proposal by making the global install command respect replace directives. Let's say we take this proposal, remove everything that mentions the replace directive and instead say "the global install command is going to respect module replace directives, directory replaces can not be supported at the moment due to a limitation of goproxy which will be addressed in a future release" and explain how it plans to support directory replaces. I'd be ok with that.

You've suggested a large change to the current interpretation of replace directives, which would break lots of existing modules.

What I have suggested is that replace directives should be respected by a global install command, I don't think this will break anything.

You've also suggested a significant change to the proxy protocol, which would be expensive to implement across the ecosystem. These are significant drawbacks, and the benefits aren't at all clear to me. It would be difficult for such a proposal to be approved.

If the proxy change can not be implemented then cloning a repository and working from there can still be implemented. That would be a shame, but what can you do? Correctness is more important than performance.

Alessandro Arzilli · Answer 50 · Sat Sep 26 2020 20:01:59 GMT+0800 (China Standard Time)

@jayconrod

I'm open to editing the proposal to discuss future modifications. The two possible paths we could take are 1) ignoring replace directives, or 2) applying only module replace directives, still reporting an error if a directory replace directive is present.

As far as I am concerned I think (2) would be a good start. At least we could focus the discussion on the merits of directory replaces rather than litigating the entire concept of replace.

Jay Conrod · Answer 51 · Wed Sep 30 2020 00:17:29 GMT+0800 (China Standard Time)

@aarzilli

Yes. This proposal would also be that proposal.

I'm not willing to consider a change to the proxy protocol in this proposal. It's very costly.

Likewise, I wouldn't consider cloning repositories. It's wildly different than what all other commands do in module mode.

What doesn't ever have merit is "use the directives of the module providing P but error if there is a replace directive" ...

I've said this earlier:

For now, we're reporting an error so there is no ambiguity about whether replace is applied or not. Additionally, starting out with strict behavior lets us relax it later without an incompatible change.

If we find that in many cases, module authors can't feasibly remove replace directives from tagged releases, then we can apply module replacements only.

If we find that replace directives aren't usually necessary in releases but are inconvenient to remove, then we can ignore them (as other commands do).

... by virtue of the fact that a global install command should not have outcomes that depend on the directory it is run in ...

I've said this earlier:

The command proposed here does not depend on the directory it is run in.

Then why are you proposing a command with a different behavior and that the current behavior of go get should be deprecated and removed?

I explained that right below the sentence you quoted.

I don't think people add replace directives to go.mod expecting them to be ignored. I think the global install command should respect replace directives so that module authors can leave replace directives in place knowing that they will be respected.

I'm sure no one would add them if they were always ignored. But they aren't. replace directives from the main module are applied, and they're useful during development.

replace directives outside the main module are ignored because they'd inevitably lead to conflicts. If the module example.com/Main requires example.com/A and example.com/B, and those modules both require and replace example.com/C with different versions, the author of example.com/Main would not be able to build anything.

I know the current behavior isn't always what people expect from replace, but I think that's largely because we haven't done an adequate job documenting it. We're making progress on that though.

I'm the author of a tool, I send a PR to one of my dependencies, unfortunately the author has taken a long vacation to Mars and the PR will be merged 6 months from now. In the meantime I fork the dependency and add a temporary replace directive. People try to install my tool using go install but it doesn't work, because replace directives aren't respected. How am I not a module user, and how isn't this taking power away from me and giving it to the Mars vacationer?

I'm interested in the specifics of situations like this. Mars is hyperbole of course, but why can't the PR be merged? Is the module permanently abandoned, in which case it's better to do a hard fork? Is the PR a bug fix or feature request? Could you work around the bug without replace in release versions until the bug is fixed? Does the author disagree with the PR? Is there a licensing reason you're unable to submit a PR?

Based on the modules I looked at when building the proposal's appendix, replace is not usually strictly necessary, or it wouldn't be necessary in the future when other proposals like #36460 land. But I think we need more data here to make a decision in either direction.

How would respecting replace directives break it?

This is explained in "Why can't module replace directives be used?".

It's because it causes inconsistencies and likely incompatibilities for modules that depend on modules providing tools. Those modules rely on replace directives in other modules being ignored.

Suppose I have a module example.com/Main that uses a static analyzer in the module example.com/analyze. I add a requirement to my go.mod, and in a build script, I have a command like:

go run example.com/analyze/cmd/analyzer -- ./...

Suppose I found a bug in a package that analyzer depends on in golang.org/x/tools, and I submitted a fix. While I'm waiting for that to be merged, I've added the line below to my go.mod:

replace golang.org/x/tools => github.com/jayconrod/tools v1.0.0-bugfix

Separately, the author of analyzer submitted a PR adding a new feature to golang.org/x/tools, so they have this replace directive in their go.mod.

replace golang.org/x/tools => github.com/analyzer/tools v1.1.0-feature

The package example.com/analyze/cmd/analyzer doesn't build without that feature, which means I can't use my go run script. My build is broken.

When I complain, the author of analyzer says, "Just use go install example.com/cmd/analyzer@latest" (i.e., this proposal except replace directives are applied). But then I have no way to build analyzer with my bug fix. Effectively, this proposal (with replace directives) took power from me, a module user, and gave it to the module author.

Jay Conrod · Answer 52 · Wed Sep 30 2020 00:34:40 GMT+0800 (China Standard Time)

@aarzilli

What doesn't ever have merit ...

... I'm not sure what other commands you think are relevant.

Just want to add, I think your tone in this discussion (and these two phrases in particular) has been unnecessarily argumentative, verging on disrepectful. Please keep the code of conduct in mind.

Jay Conrod · Answer 53 · Wed Sep 30 2020 01:54:31 GMT+0800 (China Standard Time)

@thepudds

I've added a "Future directions" section that recaps the discussion here. Please let me know if I've missed anything.

GopherBot · Answer 54 · Wed Sep 30 2020 04:45:39 GMT+0800 (China Standard Time)

Change https://golang.org/cl/258297 mentions this issue: cmd/go: error if -modfile used with 'go install pkg@version'

Russ Cox · Answer 55 · Thu Oct 01 2020 00:16:41 GMT+0800 (China Standard Time)

To summarize the discussion a bit, many questions have been asked that are now answered in the "Appendix: FAQ" in the top comment above.

The main point of contention remaining seems to be whether or not to apply replace directives. It can seem attractive to do so at first, but the result is various inconsistencies/paradoxes with go command execution and ecosystem fragmentation. My understanding is that this proposal has been discussed at the monthly tools meetings as well and that the general consensus there is that we should move ahead with replace disallowed (rejected).

Note that disallowing replace means that we can introduce semantics for replace at some point in the future. Making a decision now locks us in to something that will probably not be right. @bcmills also has some cleanup of replace planned for Go 1.17, so it would be good not to preempt that. Rejecting replace for now is the conservative behavior.

Other than replace, does anyone have any objections to accepting this proposal?

Michael Tibben · Answer 56 · Thu Oct 01 2020 13:37:36 GMT+0800 (China Standard Time)

165 used replace as a soft fork, for example, to point to a bug fix PR
instead of the original module.

242 used replace to pin a specific version of a dependency (the module
path is the same on both sides).

77 used replace to rename a dependency that was imported with another
name, for example, replacing github.com/golang/lint with the correct path,
golang.org/x/lint.

30 used replace to rename golang.org/x repos with their
github.com/golang mirrors.

11 used replace to bypass semantic import versioning.

167 used replace with k8s.io modules. Kubernetes has used replace to
bypass MVS, and dependent modules have been forced to do the same.

111 modules contained replace directives I couldn't automatically
classify. The ones I looked at seemed to mostly be forks or pins.

The modules I'm most concerned about are those that use replace as a soft fork ... other problems have other solutions that I don't think we need to design for here

I'd love to see an expansion on this - what is the recommended course of action for each of these scenarios if it's not using replace. And if this is not the place to talk about that, where might this discussion take place?

Alessandro Arzilli · Answer 57 · Thu Oct 01 2020 18:28:22 GMT+0800 (China Standard Time)

@rsc

The main point of contention remaining seems to be whether or not to apply replace directives. It can seem attractive to do so at first, but the result is various inconsistencies/paradoxes with go command execution and ecosystem fragmentation.

I think that the most important inconsistency is the one that would exist between running go build within the module that provides the package being installed and go install P. Since running build/test/etc within the context of the module that's being developed is the way programs are developed normally this would effectively introduce a different build mode (where replace directives are errors).

Russ Cox · Answer 58 · Thu Oct 01 2020 21:40:44 GMT+0800 (China Standard Time)

I think that the most important inconsistency is the one that would exist between running go build within the module that provides the package being installed and go install P. Since running build/test/etc within the context of the module that's being developed is the way programs are developed normally this would effectively introduce a different build mode (where replace directives are errors).

The different build mode effectively already exists. Today if you run "go build P" or "go install P" from any module other than P itself, the replaces are silently ignored, meaning that you get different results from the build inside the module. That different build is not what's expected for this one-shot install command; rejecting the replacements instead of silently ignoring them helps make that clear.

It seems to me that the answer is to remove any local development-only replaces before tagging a release version. The removal doesn't even have to happen on the main branch: do the remove in detached HEAD mode & tag.

Daniel Martí · Answer 59 · Thu Oct 01 2020 21:49:00 GMT+0800 (China Standard Time)

I think the disconnect here is that some people are arguing that using replace directives in releases is sometimes a necessity, but in my opinion that debate needs to happen in another thread.

Alessandro Arzilli · Answer 60 · Thu Oct 01 2020 22:00:50 GMT+0800 (China Standard Time)

@rsc

The different build mode effectively already exists. Today if you run "go build P" or "go install P" from any module other than P itself, the replaces are silently ignored, meaning that you get different results from the build inside the module.

I undestand but I don't think this is a useful or expected behavior and expanding it to the global install command would make a rare edge case into a common occourence. Why is it the case that go build P silently ignores replace directives when it's run outside of any module but in module mode?

It seems to me that the answer is to remove any local development-only replaces before tagging a release version

But what if I want my release to use replace directives?

@mvdan

I think the disconnect here is that some people are arguing that using replace directives in releases is sometimes a necessity, but in my opinion that debate needs to happen in another thread.

Absolutely agree on the first part but I don't think it this discussion can be divorced from this proposal since it amounts to a soft deprecation of replace directives.

Bryan C. Mills · Answer 61 · Thu Oct 01 2020 22:05:07 GMT+0800 (China Standard Time)

Why is it the case that go build P silently ignores replace directives when it's run outside of any module but in module mode?

That is not the case. As of Go 1.15, go build of a package from a module fails explicitly when run outside of any module but in module mode.

$ go version
go version go1.15.2 linux/amd64

$ export GO111MODULE=on

$ go env GOMOD
/dev/null

$ go build golang.org/x/tools/cmd/bundle
cannot find module providing package golang.org/x/tools/cmd/bundle: working directory is not part of a module

Bryan C. Mills · Answer 62 · Thu Oct 01 2020 22:10:10 GMT+0800 (China Standard Time)

@mtibben, the golang-nuts mailing list or the #modules channel on the Gophers Slack would be good starting points for that discussion. (At some point we should probably also write a blog post on fixing module dependencies, although the general formula in nearly all cases is “fix upstream, then upgrade to incorporate the fix”.)

Alessandro Arzilli · Answer 63 · Thu Oct 01 2020 22:22:04 GMT+0800 (China Standard Time)

That is not the case. As of Go 1.15, go build of a package from a module fails explicitly when run outside of any module but in module mode.

Interesting. If that's the case a global install command that respects replace directives wouldn't be any more inconsistent with the behavior of other commands than one that respects them but always behaves as if there was no current module. What am I missing?

Jay Conrod · Answer 64 · Thu Oct 01 2020 23:49:00 GMT+0800 (China Standard Time)

Interesting. If that's the case a global install command that respects replace directives wouldn't be any more inconsistent with the behavior of other commands than one that respects them but always behaves as if there was no current module. What am I missing?

The global install command that's currently available outside a module is go get pkg@version. go install pkg@version is meant to be similar to that. If the module named on the command line satisfies the constraints listed in the proposal, then go install pkg@version will build the same thing as go get pkg@version. The only difference is that go install pkg@version could be run from anywhere, while go get pkg@version must be run outside a module or it will use and update the go.mod in the module's directory.

So we're not introducing a new build mode. We're making an existing build mode accessible through another command, with some constraints to avoid confusion.

Russ Cox · Answer 65 · Thu Oct 08 2020 01:25:28 GMT+0800 (China Standard Time)

Based on the discussion above and the responses to my question last week, this seems like a likely accept.

Robert Clausecker · Answer 66 · Wed Oct 14 2020 00:40:42 GMT+0800 (China Standard Time)

It would be kinda cool to have similar behaviour for go test. I some times want to run benchmarks on different versions of a module to check for performance or other regressions and being able to do so without having to manually check out the repository would be great. Similarly, I believe there is a value in being able to run tests and benchmarks for dependencies (to make sure they pass) without having to check out the sources explicitly.

Russ Cox · Answer 67 · Thu Oct 15 2020 01:07:01 GMT+0800 (China Standard Time)

No change in consensus, so accepted.

Russ Cox · Answer 68 · Thu Oct 15 2020 01:07:07 GMT+0800 (China Standard Time)

(And already implemented.)

GopherBot · Answer 69 · Fri Oct 30 2020 03:50:33 GMT+0800 (China Standard Time)

Change https://golang.org/cl/266360 mentions this issue: cmd/go: print deprecation messages for 'go get' installing executables

GopherBot · Answer 70 · Wed Nov 18 2020 12:37:11 GMT+0800 (China Standard Time)

Change https://golang.org/cl/270980 mentions this issue: cmd/go/internal/modload: remove SetBuildList

GopherBot · Answer 71 · Fri Jan 22 2021 05:59:41 GMT+0800 (China Standard Time)

Change https://golang.org/cl/285452 mentions this issue: content/static/doc: document 'go install pkg@version'

GopherBot · Answer 72 · Sat Jul 03 2021 02:03:09 GMT+0800 (China Standard Time)

Change https://golang.org/cl/332569 mentions this issue: cmd/go/internal/modload: fix an apparent typo in the AutoRoot comment

Barmem · Answer 73 · Mon Apr 11 2022 20:04:37 GMT+0800 (China Standard Time)

How would i update installed executables?

Philip Whitehouse · Answer 74 · Tue Apr 26 2022 21:27:07 GMT+0800 (China Standard Time)

Why wasn't the removal of go get a major version change. What's the point of 1. if Go maintainers are happy to make breaking changes in a point release. It's fundamentally broken builds that don't use modules.

Daniel Martí · Answer 75 · Tue Apr 26 2022 21:32:27 GMT+0800 (China Standard Time)

@philipwhiuk please read https://go.dev/doc/go1compat:

Finally, the Go toolchain (compilers, linkers, build tools, and so on) is under active development and may change behavior. This means, for instance, that scripts that depend on the location and properties of the tools may be broken by a point release.

Of course they are still careful when breaking the tooling, but it's allowed by the compatibility guarantee.

Philip Whitehouse · Answer 76 · Tue Apr 26 2022 21:42:27 GMT+0800 (China Standard Time)

That "compatibility guarantee" is a big "yeah we don't care about semver for tools, good luck" isn't it.

Robert Clausecker · Answer 77 · Tue Apr 26 2022 21:47:59 GMT+0800 (China Standard Time)

@philipwhiuk Again, the compatibility promise is about the language and libraries, not the tooling. It's more about being able to write code that will compile in the future. The toolchain is disposable and can change, the language is stable.

Xe Iaso · Answer 78 · Tue Apr 26 2022 22:03:28 GMT+0800 (China Standard Time)

I know this is probably the wrong place to put this and that this feedback is likely going to be ignored, but this whole "the tooling is exempt from the compatibility promise" thing kind of feels like a cop-out. I understand the intent of this, as you use Go in production places then you will learn more things and realize what mistakes were made; however the community starts to standardize and expect the behaviour of commands to stay consistent unless there is a major version change. This violation of expectations is why you see people make more heated or angry comments. It feels like the tool changed out from under them and now they have to re-learn the tool.

I'd be willing to argue that the semantics of how commands like go get work is probably more important than the standard library being stable. Something like go fix can correct breaking changes in the standard library, but there is no go fix to correct the changes in how people install commands or the documentation people have written about how to install various tools written in Go.

Again, I get the point of why the tooling isn't behind the compat promise. It allows you to make the tooling better and fix behaviours that were outright bugs, but overall I'm really not sure if continuing to keep Go at 1.x post-modules was really a good idea. I know that politically making a "Go 2" is a difficult thing, but the kinds of changes that modules have given developers at nearly every level of the stack really seems like it should have been a candidate for being released as "Go 2". It's sad that this is politically difficult though, because "Go 2" has been lauded as a magical fairy unicorn thing where the standard library is going to be fixed forever or whatever. If anything what we have now may as well be "Go 2" because the Go I started learning near a decade ago is almost a completely different language than the Go I use today.

Philip Whitehouse · Answer 79 · Tue Apr 26 2022 22:08:00 GMT+0800 (China Standard Time)

@philipwhiuk Again, the compatibility promise is about the language and libraries, not the tooling. It's more about being able to write code that will compile in the future. The toolchain is disposable and can change, the language is stable.

It barely matters if the code still "compiles" if I can't actually get it to compile by running the same commands. Code that works on 1.13.4 will have to be rewritten to use modules most likely because there's no explanation of how to do what I was doing with go get in modern Golang. The compile promise is worthless on it's own.

Daniel Martí · Answer 80 · Tue Apr 26 2022 22:13:29 GMT+0800 (China Standard Time)

I think we're getting off topic for this thread, especially as it's been closed for a while. I would suggest to give #37755 a read, which was a proposal to actually keep the pre-modules workflow working forever. For any other meta discussion about the backwards compatibility guarantee, I would personally suggest https://groups.google.com/g/golang-dev or perhaps a new issue if you have a specific proposal in mind.

mihaiav · Answer 81 · Mon May 02 2022 10:08:33 GMT+0800 (China Standard Time)

After I've "upgraded" to go 1.18 another go command (this time go install) beame totally useless and my all my builds fail. All the related issues seem to be closed.
Better remove all the go commands so that can go back to good old make files. At least we know what we are dealing with.

Brian Topping · Answer 82 · Thu Nov 17 2022 10:53:28 GMT+0800 (China Standard Time)

I was led here via https://golang.org/doc/go-get-install-deprecation.

Didn't realize this was going on at all. Whether it was sloppy/lazy or whatnot, I would typically check out sources on projects I wanted in GOPATH via GOMODULE111=off go get XXX. A one-liner that checked out the sources. Reading around today, I see that I should have been using go get -d. Okay.

Created a new machine today and blindly installed the latest. My muscle memory doesn't work. Wasted a couple hours to realize it wasn't because some magic was lost in my GOPATH, the feature was removed.

Is there a new way to populate $GOPATH/src without creating directories, changing to them and manual git clone?

I guess I always considered that a feature and not an edge case?

Daniel Martí · Answer 83 · Thu Nov 17 2022 16:42:44 GMT+0800 (China Standard Time)

@briantopping #31529 tracks what would be the replacement for GO111MODULE=off go get in terms of VCS cloning into a useful directory.

fubss · Answer 84 · Tue Apr 04 2023 18:38:43 GMT+0800 (China Standard Time)

For those who like me:

need to upload the source code of the project to the ~/go/src directory;
are not comfortable with git clone;

install an additional version of go where else one could use go get (I installed 1.18.10):
1.1) go install golang.org/dl/go1.18.10@latest
1.2) ~/go/bin/go1.18.10 download
1.3) ~/go/bin/go1.18.10 version
export GO111MODULE=off
~/go/bin/go1.18.10 get -d [desired_project_repo_here]

cmd/go: 'go install' should install executables in module mode outside a module

Abstract

Background

Proposal

Details

Examples

Current go install and go get functionality

Rationale

Why can't go get clone a git repository and build from there?

Why can't vendor directories be used?

Why can't directory replace directives be used?

Why can't module replace directives be used?

Why report errors instead of ignoring replace?

Should go.sum files be checked?

Why require a version suffix when outside a module?

Why not a -g flag instead of @latest?

Compatibility

Implementation

Future directions

Appendix: FAQ

Why not apply replace directives from all modules?

How can module authors avoid replace?

When would go install be reproducible?

What happens if a module depends on a newer version of itself?

Appendix: usage of replace directives

Current `go install` and `go get` functionality

Why can't `go get` clone a git repository and build from there?

Why can't directory `replace` directives be used?

Why can't module `replace` directives be used?

Why report errors instead of ignoring `replace`?

Should `go.sum` files be checked?

Why not a `-g` flag instead of `@latest`?

Why not apply `replace` directives from all modules?

How can module authors avoid `replace`?

When would `go install` be reproducible?