Giters

golang / go

The Go programming language

Home Page:https://go.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

cmd/go: clarify best practice for tool dependencies

myitcv opened this issue · comments

commented

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64
vgo commit 22e23900224f03be49670113d5781e4d89090f45

Does this issue reproduce with the latest release?

Yes; and latest vgo commit (per above)

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/tmp/tmp.VQw1O3x8Wy/hello/bin"
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.VQw1O3x8Wy"
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build414355570=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Off the back of #25624 (comment), I'd like to confirm that the following represents the "best practice" advice when adding and installing tool dependencies:

cd `mktemp -d`
export GOPATH=$PWD

mkdir hello
cd hello
vgo mod -init -module example.com/hello

# this could be anywhere but for convenience...
export GOBIN=$PWD/bin

# add a dependency on golang.org/x/tools/cmd/stringer
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

vgo install golang.org/x/tools/cmd/stringer

The go.mod and .Target for stringer look fine:

$ cat go.mod
module example.com/hello

require golang.org/x/tools v0.0.0-20180615195736-465e6f399236
$ vgo list -f "{{.Target}}" golang.org/x/tools/cmd/stringer
/tmp/tmp.VQw1O3x8Wy/hello/bin/stringer

The issue however is that running vgo mod -sync then removes our module requirement on golang.org/x/tools - I suspect this is a bug:

$ vgo mod -json
{
        "Module": {
                "Path": "example.com/hello",
                "Version": ""
        },
        "Require": [
                {
                        "Path": "golang.org/x/tools",
                        "Version": "v0.0.0-20180615195736-465e6f399236"
                }
        ],
        "Exclude": null,
        "Replace": null
}
$ vgo mod -sync
warning: "ALL" matched no packages
$ vgo mod -json
{
        "Module": {
                "Path": "example.com/hello",
                "Version": ""
        },
        "Require": null,
        "Exclude": null,
        "Replace": null
}

If we assume this is a bug and ignore it for now, I also wonder whether we can improve this workflow for adding tool dependencies somehow. The following steps feel a bit "boilerplate" and unnecessary:

  • define a fake tools.go file
  • add a dependency on the tool (which will never compile because it's a main package, so I'm not sure we can ever safely verify tools.go is "good"?)
  • set GOBIN to an appropriate location
  • vgo install tool
  • ensure PATH includes GOBIN

I wonder whether we could in fact obviate all of this by having something like:

vgo install -tool golang.org/x/tools/cmd/stringer
vgo run golang.org/x/tools/cmd/stringer

Thoughts?

vgo run tool is possible as a result of #22726, but because of #25416 it effectively requires a link step each time.

What did you expect to see?

With respect to what I think is a bug with vgo mod -sync

go.mod unchanged by the vgo mod -sync

What did you see instead?

The golang.org/x/tools requirement removed.

/cc @rsc @bcmills

commented

Let's take a step back: what do you mean by “tool dependency”?

The stringer example suggests that you mean “a tool required for use with go generate”, but note that generate is intended to be run by the author of the module, not its users: a tool needed to run generate shouldn't be a hard requirement for the module or its tests.

commented

Let's take a step back: what do you mean by “tool dependency”?

Roughly:

  • I'm writing a module; or, more generally, me and my team are writing a module
  • we want to use a tool (e.g. stringer) whilst writing that code
  • we want to reproducibly re-generate code, or put another way, we want the whole team to consistently generate the same code at any given point in time
  • hence I want to ensure that everyone is using the same version of any tool we require

but note that generate is intended to be run by the author of our module, not its users: a tool needed to run generate shouldn't be a hard requirement for the module or its tests.

Correct, this would only be a dependency for the authors of the module (as you point out, the users of our module probably don't care whether or not we've used stringer or other tools).

As I understand it, these tool dependencies do not introduce any additional requirements for users of our module: they (the tool dependencies) can only ever be referenced by running vgo install or vgo run in the context of our module. So as a user of our module, I would, by definition be in a different context (specifically the module that references our module) when running vgo install or vgo run.

I'm using the term "context" here to mean "the go.mod that vgo uses." I'm sure there must be a better, more precise term

commented

I haven't played around with vgo much yet or looked under the hood.

I know a lot of package managers handle this by sorting dependencies into buckets: regular dependencies, test dependencies, dev dependencies. That seems like overkill and there can also be dependencies by build tag, like, say, importing a module for internal use but only on windows, and dependencies may only be for a certain package in a module that you may not be using.

Could vgo record all the dependencies for a project—for all build tags, for all packages, for tests, for development—but lazily fetch them as needed? (This may require more invasive integration but it would be an optimization as the semantics are the same for lazy-fetch and fetch-everything-upfront)

commented

@jimmyfrasche

Could vgo record all the dependencies for a project—for all build tags, for all packages, for tests, for development—but lazily fetch them as needed?

My understanding is that this is exactly what vgo does today, for any dependency, test/tool or otherwise.

And, to your earlier point, I agree it means the delineation between the buckets is unnecessary (for anyone wanting to know why a dependency exists, vgo can tell us that answer).

Just to clarify one point above. My proposal for vgo install -tool golang.org/x/tools/cmd/stringer would, however, add a section to go.mod (or equivalent) where the tool packages are listed. This is simply to avoid the need for the tools.go file referenced in my example.

commented

A couple of observations:

  1. Some tools required by Go builds are not themselves built using the go tool. (For example, Google's protocol compiler is written in C++; only the Go code generator plugin is written in Go.) So even if we add support for tooling to the go tool, it will be at best a partial solution.

  2. If you are assuming the use of the go tool anyway, then you can write a Go program that invokes it with appropriate arguments.

  3. Because a command-line tool is a top-level package, its versions are not context-sensitive: if you're not mixing in other packages to customize the tool, you don't need to resolve version constraints from those packages.

To me, those suggest a fairly straightforward solution: go run pkg@version. In combination with //go:generate, that would allow you to express the dependency (at the exact version used) directly in the source file:

//go:generate go run golang.org/x/tools/cmd/stringer@v1.11.0 -type=Pill

In contrast, if you are building a custom version of the tool, then you should be able to customize it using blank-imports as you describe.

commented

@bcmills that sounds good but it would be awkward to write that each time and easy for things to go out of sync and you have some code using 1.11.0 and some using 1.12.3 but probably want .

go generate let's you define aliases with //go:generate -command alias the-command (I've not used it personally and unsure of the scope: file? package? (I assume file.))

Perhaps a more vgo-aware variant would be helpful, like

//go:generate -require golang.org/x/tools/cmd/stringer@v1.11.0

which is module scoped. It falls somewhere between an ephemeral go install -tool and a fictitious blank import.

There are a some details to iron out in such an approach, of course.

commented

@bcmills

Some tools required by Go builds are not themselves built using the go tool.

Completely. As you say, this can either be solved entirely orthogonally, or with Go wrappers (wrappers that could even take care of installing the right version).

Because a command-line tool is a top-level package, its versions are not context-sensitive: if you're not mixing in other packages to customize the tool, you don't need to resolve version constraints from those packages.

I'm not entirely sure I follow your point here. The point I was trying to make is that where (i.e. in what directory) you run vgo install golang.org/x/tools/cmd/stringer is relevant because that will determine which go.mod you reference and therefore which version of the golang.org/x/tools module you use.

To me, those suggest a fairly straightforward solution: go run pkg@version.

As @jimmyfrasche pointed out this has the major downside of littering the required version across all the places you require go:generate directives, which seems to be somewhat counter to having go.mod be the single point of reference for versions.

Just one point to note about the "proposal":

vgo install -tool golang.org/x/tools/cmd/stringer
vgo run golang.org/x/tools/cmd/stringer

Despite the explanation in #25416, I think vgo run golang.org/x/tools/cmd/stringer could actually cache the linked binary, because the cache can very precisely be trimmed given we know exactly what version is being referenced at a given point in time.

commented

Running a tool at a specific version depending on the module would have to go through vgo.

Running stringer lets $SHELL pick which stringer to run, so it would need to be something like go run-dev-tool stringer so that vgo could dispatch the correct version of the command.

Something like that could be stuffed into go run but that doesn't seem to fit with the spirit of that command. Arguably go tool would be a better fit but that could cause namespacing issues so it seems like a separate command built to purpose would be best.

go generate could handle this transparently but not all tools are necessarily for code generation. You may also want a specific version of, say, a linter.

An IDE/editor could use tool dependencies to figure out if this is a tool dependency or a regular program and run it through vgo if need be. From the command line, though, it would be up to the user to know whether to run stringer vs go run-dev-tool stringer so there would still be potential skew with the version documented. I don't think there's much to do about that, though.

A tool module could have multiple commands. Look up would be slow if these weren't included somewhere in the metadata.

A tool module could have multiple dependencies, some shared with the module under development. Unlike with other dependencies, you probably don't want those to be solved together: using foobar shouldn't affect your version baz or the version of baz used by other tools you depend on. They should all be siloed.

This is all starting to sound mostly unrelated to vgo but perhaps it's something that can be built on top of vgo.

Going back on what I said about buckets, let's say there was as separate tool godev and a separate file dev.mod.

godev install foo updates dev.mod recording the version and the exposed commands. Instead of installing the command globally it installs the binaries outside of $PATH but somewhere it can easily grab the correct version. It updates dev.mod with the vgo style import and version path but also records the names of the commands. For the most part it's a wrapper around go get that does some extra work.

godev run foo invokes the appropriate foo from dev.mod.

Is there anything that would need to change in vgo to allow something like this?

A downside of not having it integrated in vgo itself would be that go:generate directives would need to explicitly prefix godev run.

commented

What is the decision to be made here?
I think the tools.go file is in fact the best practice for tool dependencies, certainly for Go 1.11.
I like it because it does not introduce new mechanisms.
It simply reuses existing ones.

commented

I think the tools.go file is in fact the best practice for tool dependencies, certainly for Go 1.11.

Indeed, we don't need to "solve" this now.

What is the decision to be made here?

For Go 1.11

  • is the vgo mod -sync behaviour observed in the description a bug or not? It feels like it is.

For later:

  • is there a better way to ensure the fake tools.go is valid? Because it will never compile. We could do something with vgo list and a bit of shell, but it feels like we're side stepping the issue
  • is there a way to reduce the "boilerplate" setup described in the issue, to make the experience for module maintainers nicer?
commented

Adding another comment to this thread to give the equivalent, and current, go commands

This is a go-based rewrite of #25922 (comment)

cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# Either rely on GOBIN=GOPATH/bin or set it explicitly
export GOBIN=$PWD/bin

# add a dependency on golang.org/x/tools/cmd/stringer
# the build constraint ensures this file is ignored
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

go install golang.org/x/tools/cmd/stringer

cat go.mod

results in:

module example.com/hello

require golang.org/x/tools v0.0.0-20180813205110-a434f64ace81 // indirect
commented

Similarly, converting an existing project from dep to the modules system and running go mod vendor results in the stringer tool and its immediate dependencies being removed from the vendor directory.

Sorry, this is actually working. My tools.go file was missing the package directive at the top of the file and because it's never built the compiler never got a chance to complain. This, ironically, is one of @myitcv's concerns in his initial comment.

commented

Best practice remains creating the tools.go file.
It's true that go mod init does not auto-create a tools.go from dep's config,
but I think doing so is getting a bit beyond scope.

commented

Probably it's worth to highlight why tools.go should contain some build tag like tools (but not ignore): to avoid side-effects of init() functions in tool dependencies.

commented

A related comment: as far as I am aware, creating a tools.go as described above in #25922 (comment) seems to work with go mod vendor for Go-based tool dependencies.

In addition, there is also this separate comment in #26244 (comment) :

vgo vendor (now vgo mod -vendor or go mod -vendor) does not copy entire repositories. By design, it copies only the directories needed to satisfy imports in builds of the top-level module. If your module is not importing github.com/antlr/grammars-v4 then that directory will not be copied. And if there are directories with only non-Go files, there is no way to copy those directories into vendor. That's not what vendor is for. Vendor is only about preserving the code needed to build the main module

...which on the one hand seems to say go mod vendor is not intended to cover all use cases around non-Go pieces of a repository...

...but that comment also seems to suggest that if there is a piece of Go code that will be vendored, the operation seems to function at the granularity of directories (e.g., it "copies only the directories needed to satisfy imports in builds of the top-level module").

In practice, this currently seems to mean that non-Go tool dependencies (such as a shell script) are copied into the vendor directory by go mod vendor if they are in the same directory as a vendored piece of Go code. So a tools.go-based approach could in theory be extended to pulling in non-Go based tool dependencies (which would happen naturally if the non-Go tools are intermixed at the directory level with Go code, or perhaps even by adding something like a vendorme.go to a particular directory if needed).

However, as far as I can see, the go mod vendor documentation currently does not say it operates at the directory level, so perhaps that is behavior that should not be relied upon, or perhaps this would otherwise be considered an undesirable approach.

EDIT: note that a fair amount of care is required if you are pulling in additional directories beyond what go mod vendor does naturally, and this is less about vendoring behavior and more about following the go tool's build model. See for example comment from Russ in #26366 (comment), as well as https://golang.org/cl/125297, and #26366 (comment))

commented

I've tried the solution proposed by @myitcv, it more or less works, seems like stringer can't solve the dependencies in the package that has the //go:generate:

~/Code/goreleaser/goreleaser vgo*
λ go list -f "{{.Target}}" golang.org/x/tools/cmd/stringer
/Users/carlos/Code/goreleaser/goreleaser/bin/stringer

~/Code/goreleaser/goreleaser vgo*
λ go generate ./...
stringer: checking package: artifact.go:11:2: could not import github.com/apex/log (type-checking package "github.com/apex/log" failed (/Users/carlos/go/pkg/mod/github.com/apex/log@v0.0.0-20180702155952-941dea75d3eb/stack.go:3:8: could not import github.com/pkg/errors (cannot find package "github.com/pkg/errors" in any of:
	/usr/local/Cellar/go/1.11/libexec/src/github.com/pkg/errors (from $GOROOT)
	/Users/carlos/go/src/github.com/pkg/errors (from $GOPATH))))
internal/artifact/artifact.go:15: running "stringer": exit status 1

Not sure if it is a bug in stringer or somewhere else, happy to provide more info if needed though.

Not sure if useful, but:

commented

@caarlos0 - this is because stringer requires code to type-check in order to work. And in your situation it can't type check because it can't resolve an import. stringer can't resolve an import because it doesn't "know" about modules.

stringer is being fixed to work with modules; this work is being tracked under the umbrella of #24661

commented

I'm fairly sure I'm not in favour of mixing tool dependencies in with all the other project dependencies. I think it's good to have reproducible tool behaviour ("I used vX.Y.Z of tool T" should mean something), and attaching tools to arbitrary projects' go.mod files will mean that the tool binaries will be built with arbitrary (although notionally compatible) dependencies. In my view, when we use a tool to build parts of a project, we don't necessarily wish that tool to be part of the project, merely that it's a way of creating parts of that project.

Part of the promise of MVS is that of reliable and predictable results. That predictability applies to everything below a given module root, but it doesn't necessarily apply when a module is included inside a module root.

If there's anywhere that we want to be predictable, it's the building of binaries, so when building external tool binaries, I think we should respect exclude and replace clauses in their top level module, and use exactly the versions that are implied by that top level module, not the top level module that we're building the tool for.

If we do this, it means that when someone raises an issue about the behaviour of a tool, they can say "this went wrong with v3.1.4" and the tool maintainer can know exactly what binary they're using.

So I propose that we have another file, say go.tools, that lists required Go-implemented tool packages and their versions.

For example:

 golang.org/x/tools/cmd/stringer v1.0.3
 gopkg.in/httprequest.v1/cmd/httprequest-generate-client v1.2.0

I don't think it should be in go.mod itself, because these tools are explicitly not governed by the dependencies in go.mod.

One problem with this is that it doesn't work well when the tool doesn't have a go.mod file. In that case, we could fall back to using the local go.mod file ( the tools.go approach above could work). If a go.tools entry doesn't have a go.mod file and it's not in the dependencies, then the build of that tool would fail. This is only necessary while there are Go projects that don't contain a go.mod file.

Another question is what to do about dependency updates? Currently the go tool manages dependency updates very nicely. One possible idea is that we could treat go get of a main package as an explicit request to add the command to the go.tools file.

Then go get golang.org/x/tools/cmd/stringer@latest would update the version in go.tools to the latest version. It could also make the binary available for use by go run too.

Thoughts?

commented

when building external tool binaries, I think we should respect exclude and replace clauses in their top level module, and use exactly the versions that are implied by that top level module, not the top level module that we're building the tool for.

I can see the value in that, but I think there is also substantial value in being able to, say, fix a bug in the dependencies of a tool and apply that fix immediately (e.g., using a local replace directive).

If a tool vendor wants builds to be perfectly reproducible, they need the user to replicate their configuration exactly: not just the same dependencies, but also the same version of the compiler, same experiment flags, same gcc, etc. I don't think that merely pinning the go.mod will really solve that: it's a distribution problem, not (just) a reproducibility problem.

At any rate, it's easy enough for users to include the actual configuration when reporting problems by attaching the output of go env and goversion -m.

commented

I can see the value in that, but I think there is also substantial value in being able to, say, fix a bug in the dependencies of a tool and apply that fix immediately (e.g., using a local replace directive).

I can see the value in that too. I think there's value in allowing both approaches. For example, the go.tools file could allow us to specify that a given tool is governed by our local go.mod file.

I'm also concerned about tool dependencies leaking through into our production code dependencies. We already have that kind of bleed-through from external test dependencies, but the tools are even less part of the code, and some can potentially have large and tricky dependency graphs. Isolating those from the main project dependencies seems like it would be a good thing to me. I'd like to be able to use the latest version of a tool without necessarily pulling all my dependencies forward at the same rate.

As someone that struggles daily with near-intractable diamond-dependency issues, keeping some isolation between components that are potentially entirely independent seems like something that's good to aim for. There's no need for a separate binary to use all the same versions of all its dependencies as the main code, and there are significant advantages in allowing them to be different.

commented

I'm also concerned about tool dependencies leaking through into our production code dependencies.

That's fair, but that seems like a better role for go list -deps than the go.mod file itself: the go.mod file is pretty much always an approximation (due to the granularity of modules being larger than packages).

I'd like to be able to use the latest version of a tool without necessarily pulling all my dependencies forward at the same rate.

IMO that's the more interesting angle, but there is also an empirical question buried in there: how much harm does/will it cause to have tools pull your other dependencies forward, compared to the rate at which you would normally upgrade those dependencies anyway?

commented

IMO that's the more interesting angle, but there is also an empirical question buried in there: how much harm does/will it cause to have tools pull your other dependencies forward, compared to the rate at which you would normally upgrade those dependencies anyway?

FWIW it applies the other way too: I'd like to be able to continue to use an old and stable version of a tool while I roll dependencies forward on my project willy-nilly.

commented

It seems that I found a case when having the tools package is not really helpful.

I use goconvey in some projects.
When I define it as a "tooling dependency", go mod vendor stores only *.go-files. But for this particular tool I need some UI-relevant static files as well. As I understand, there is not any way to tell go mod to "vendor" non-go files. Am I right?

commented

I've also tried to implement a tools.go-based approach as a tiny go mod / dep command.
https://github.com/izumin5210/gex

I used it in some projects(e.g. wraperr, grapi, ...).
And I found some cons of this approach:

go.mod (Gopkg.toml) contains tool dependencies.
When a project depends on another package that has tools.go , the project's go.mod also contains tools.
If this approach is implemented in go mod, we will think that we should split a manifest file(e.g. go.mod and go-tools.mod).

commented

I made a related issue here: #29494 In my case it's about a hot reloader tool for development. Some parts of this discussion seem to assume this is only about code generators like stringer, but it applies to linters, hot reloader tools, or any other go module that I don't want compiled up into the application executable, but instead want to use from the command line before the application starts (or even the tool that does the starting-up of the application)

I like the go install -tool option as a UI and the go.tools file suggested by @rogpeppe — that seems explicit and clear.

commented

We are running into a similar issue. In our Org, we aren't allowed to download dependencies from the internet during CI. Historically this would require us to Fork/Mirror a repo and bring it in house. But with go modules, we can just vendor the deps and check them into source control. That way, our CI servers aren't trying to pull packages/code from github/golang.org/google.golang.org etc.

This approach works perfectly for packages that are actually being used in the code, but for build/tooling packages, it falls a little short. I really like what @tsheaff and @rogpeppe laid out with go install -tool and the go.tools file. Being able to explicitly declare a package as a tool/build dependency would be very nice. And ideally, that code would not be a part of the final built binary, just used during the build/test/cover steps (i.e. CI/CD process).

commented

Best practice remains creating the tools.go file.

I've failed to get that working:

tools.go

// +build tools

package sdm630

import (
	_ "github.com/mjibson/esc"
)

go.mod

module github.com/gonium/gosdm630

require (
	github.com/mjibson/esc v0.1.0 // indirect
)

being used by

//go:generate esc -private -o assets.go -pkg sdm630 -prefix assets assets

results in

❯ go generate
http.go:21: running "esc": exec: "esc": executable file not found in $PATH
commented

@andig you need to install the tool first, per https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md.

@tjsampson - you can use the vendor directory for tool dependencies too, but you need to tell the go tool to use the vendor when installing the tool. So in the link I provided above, instead of go install your.com/tool you would do go install -mod=vendor your.com/tool.

@tsheaff - there is nothing to prevent any arbitrary tool from being added as a dependency via the mechanism described here. I have a file watcher added as one of my tools for example. The approach is the same.

If however you definitely want to "globally" install a tool, there is support for that in the beta release of Go 1.12 (added in response to #24250) or you could consider github.com/myitcv/gobin (example)

commented

@andig you need to install the tool first, per https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md.

I'm probably missing something here, but when using build-only dependencies without runtime use like https://github.com/mjibson/esc or stringer, what is the point of go.mod'ing them if they still need to be installed? Only selecting the specific version?

commented

Only selecting the specific version?

Correct.

To avoid installing you can modify your //go:generate directive to something like:

//go:generate go run golang.org/x/tools/cmd/stringer ARGS

or else for faster execution use gobin:

//go:generate gobin -m -run golang.org/x/tools/cmd/stringer ARGS

(the reason for using gobin over go run is covered in the gobin wiki)

commented

I notice that the tools.go approach is now documented at http://golang.org/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module.

commented

go.mod (Gopkg.toml) contains tool dependencies.
When a project depends on another package that has tools.go , the project's go.mod also contains tools.
If this approach is implemented in go mod, we will think that we should split a manifest file(e.g. go.mod and go-tools.mod).

@izumin5210 is there an open issue for this? I couldn't fine one.

EDIT: looks like myitcv/gobin#44 contains some discussion

commented

tools.go pattern worked for me too, but I've noticed that go mod tidy removes all dependencies recorded in tools.go from go.mod & go.sum, which doesn't seem to be mentioned in this thread.

commented

@prymitive Just tested and that does not happen for me, on Go 1.12

commented

@tv42 sorry, my bad, I had an incorrectly formatted comment in my tools.go so it wasn't working and because it had // +build tools I didn't get any compile / mod tidy errors. Fixing the comment fixed tidying.

commented

Copying / linking to a StackOverflow post I made here because when I google to find it again I end up back on this thread pretty much every time, thinking it's the place I put the answer... and so now it will be :)

How do go modules work with installable commands?

(original SO post ^^)

Using a "tools" package

I made a tools directory:

mkdir -p tools

I put the tools package inside of it (as mentioned above):

// +build tools

package tools

import (
    _ "github.com/UnnoTed/fileb0x"
)

Note that the tag is mostly not important. You could use foo:

// +build foo

However, you cannot use ignore. That's a special predefined tag.

// +build ignore

// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies

Updating go.mod

The best way is probably to run go mod tidy:

go mod tidy

There are a number of commands that affect go.mod, some of which I had run before go mod tidy:

go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor

However, later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.

vendoring

In order to actually take advantage of the modules cache in a project you need to copy-in the source code

go mod vendor

That will grab all dependencies from go.mod

You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.

go fmt -mod=vendor ./...
go generate -mod=vendor ./...
go build -mod=vendor ./...

That includes go fmt, go build, go get, go install, and any go run called by go generate (and even the go generate itself)

//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main

// ...

If you get an error

I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:

tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package

(fileb0x is the thing I'm trying to add)

That's a result of not having the proper //+build tools comment

commented

@solderjs - regarding ignore being special: #29598

I'm not clear what the rest of your comment is referring to. Is there a problem with #25922 (comment) as written?

commented

@myitcv No problem with the way it's written. I just compiled all of the gotchas that I've run into together and pasted them in here so that next time I'm searching for it and this pop ups on top of the results, I'll have cumulative answer right where I want it.

commented

@solderjs understood.

What problem were you trying to solve by using go mod vendor? Because, at least in my experience, I can't think of a problem that requires go mod vendor as a solution, unless you're simply looking to provide vendor as, well, a vendor so that users (not developers of the module) of your code then don't have to also fetch dependencies. The instructions for go fmt etc are what's confusing me, because if you were a developer of said module you wouldn't choose to use the vendor, instead you'd rely on the module cache.

commented

Note to those still getting the error:

tools/tools.go:5:2: import "golang.org/x/tools/cmd/stringer" is a program, not an importable package

It turns out that you must have an empty line between the build tag and the package statement.

Bad

// +build tools
package tools

Good

// +build tools

package tools

@rsc is this by design?

commented

@shomron

is this by design?

Yes it is: https://godoc.org/go/build#hdr-Build_Constraints

To distinguish build constraints from package documentation, a series of build constraints must be followed by a blank line.

commented

@rsc following up on your last comment in this issue, will the tools.go workaround be the official way to version main packages inside a go.mod file forever?

I think it would be great if Go had official support for versioned main packages inside go.mod and maybe provided a way to execute versioned binaries as well.

I'd like to open an issue but I just wanted to check first if the tools.go workaround will be the official answer even after Go Modules becomes on by default.

Thanks!

commented

My impression was that tools.go is the favored solution now, but the whole concept is still very experimental and not widely acknowledged.

commented

@marwan-at-work - just to mention that on the golang-tools calls over the last few months, and in #30515, @ianthehat has expressed interest in solving this and the broader issue of "tooling". That culminated in #30515 (comment), and most recently a mention on the April call in which @ianthehat suggested gobin's approach might well serve as a useful straw man starting point.

All that said, I think it's probably worth either opening a new issue with your question, or asking over in #30515 - because comments in closed issues are easily missed.

commented

I found my way to carry and install tools as code dependencies. I've tried to describe it here https://github.com/nordicdyno/golang-tools. I hope it could be handy for anybody who is looking for a solution on how to carry tools with project code and don't abuse its dependencies list (and versions) by tools dependencies.

In short: I put dependencies to hidden from go build tools directory (with _) as a separated module with dependencies and manage all such tools and its dependencies independently.

commented

@nordicdyno thanks! @rsc 1.13 has made the tools.go approach obsolete, any word on this? My use case is wanting to include https://github.com/golangci/golangci-lint and https://github.com/git-chglog/git-chglog for development usage only.

From https://github.com/nordicdyno/golang-tools:

Before Go 1.13 solution with tools.go (#25922) had worked well enough without abusing go.mod/go.sum file by dependencies hidden under a tag. Now with go 1.13.x go mod tidy adds such dependencies to module's go.mod go.sum files, which I want to avoid.

commented

@montanaflynn, this issue is closed, and to my knowledge there was no change in actual behavior in Go 1.13 — just a decision about best-practices.

If you believe that there was a regression in behavior in Go 1.13, please open a new issue with steps to reproduce it.

commented

The best-practice examples above leave it a mystery (at least to me) how to get reproducibility – meaning it is not clear how someone who is not the author of tools.go would get the same version of tools used by go:generate on their path.

I know the issue is closed, and perhaps the best practice is clear to others. But I'm struggling to find a complete example that demonstrates how a team of people can work with the same tool dependencies (e.g. run the same code in go generate).

commented

@tschaub if you haven’t already, be sure to take a look at this working example, which includes using go generate:

https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md

commented

Thanks, @thepudds. That is a nice example demonstrating what the original author might do. But unless I'm mistaken, it doesn't cover how someone else working on the same project might get the same.

That is, after the original author creates the module,go installs dependencies, and alters their path, what steps should someone follow after git clone if they wanted go generate to produce the same?

commented

@tschaub there is no standard way to do that with go, however there is an additional tool you can introduce to sort of get that outcome.

Its called gobin and its use is described in https://github.com/go-modules-by-example/index/tree/master/017_using_gobin

This makes the minimum common set of dependencies for your team simply go + gobin, and gobin's installation can be automated with a makefile.

Here is an example I set up on the netlify open-api repo when I worked there:

https://github.com/netlify/open-api/blob/813b6ad88723e32e8c6760c0a28dfd7a254d2199/Makefile#L13-L14

and the accompanying generate file:

https://github.com/netlify/open-api/blob/813b6ad88723e32e8c6760c0a28dfd7a254d2199/generate.go#L3

This also takes advantage of the tools.go pattern.

I would love to see this practice standardized however it doesn't look like it will happen anytime soon.

commented

Since this is showing up in my searches when I'm trying to find the other reference that I made, I'm going to add it here as well:

If you get an error

I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:

tools/tools.go:6:5: import "git.rootprojects.org/root/go-gitver" is a program, not an importable package

(go-gitver is the thing I'm trying to add)

I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:

Using a "tools" package

I made a tools directory:

mkdir -p tools

I put the tools package inside of it (as mentioned above):

// +build tools

package tools

import (
    _ "git.rootprojects.org/root/go-gitver"
)

Note that the tag is mostly not important. You could use foo:

// +build foo

However, you cannot use ignore. That's a special predefined tag.

// +build ignore

// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies

Updating go.mod

The best way is probably to run go mod tidy:

go mod tidy

However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:

go install git.rootprojects.org/root/go-gitver # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor

Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.

vendoring

In order to actually take advantage of the modules cache in a project you need to copy-in the source code

go mod vendor

That will grab all dependencies from go.mod

You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.

go fmt -mod=vendor ./...  # this needs a fix that is scheduled for go1.15 I believe
go generate -mod=vendor ./...
go build -mod=vendor ./...

That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)

//go:generate go run -mod=vendor git.rootprojects.org/root/go-gitver --fail
package main

// ...

Reference

From https://stackoverflow.com/a/54028731/151312

commented

Here is a modification to the best-practices example that makes things reproducible (without additional installs or path modifications) on Go 1.13.8: https://gist.github.com/tschaub/66f5feb20ae1b5166e9fe928c5cba5e4

tl;dr - use go run golang.org/x/tools/cmd/stringer instead of stringer in your go:generate comment.

Update: I see now that this same suggestion appears above (#25922 (comment))

commented

@tv42 What needs improvement in my example? As far as I can tell I've collected the best information from various threads and combined them into a single set of "just works" instructions, but I'd love to continue to improve it.

commented

Really like the

go run tool@version

idea.

But on Go 1.14, only go get supports the @version syntax.

Is there a way run or install a specific version of a Go executable in Go 1.14?

commented

Hi Awesome Go Community! 👋

Thanks for this thread @myitcv and everyone else who contributed to this discussion. Maybe it's not properly appreciated but a huge amount of people traverse through those threads carefully every day to find answers, so big kudos to you all. Especially this thread, for us, was extremely insightful and gave us many pointers in the space of tools versioning.

This being said, we are maintaining several big(-ish) Go projects in the open-source (e.g Prometheus, Thanos, prometheus-operator and more) and we were looking for a solid solution to this problem for a few months already. While there are good experiments, we were looking for something we can rely on now and long-term.

This is why we created the open-source CLI called bingo, that (hopefully) makes it completely seamless. 💪

So, If you are looking for some smooth experience for versioning Go tools on top Go Modules for any project (even non Go projects that just use Go tools!) I would recommend taking a look and trying out! 🤗 Feel free to use and please contribute & give us feedback! We started to use it recently in many projects we maintain e.g Thanos and so far it works quite well... 🚀 Hopefully we can maintain it together and improve on the way.

After all, even if Go Team will have an answer to this problem someday (I am pretty sure they will!), I think it's only helpful if the community can try to solve this problems much earlier on top of existing tooling (Go Modules). I think it might be good experience for Go Team as well to tell what requirements matter here. 🤗

I also wrote some detailed blog post on the way about this problem space and a little bit on how bingo was created: https://www.bwplotka.dev/2020/bingo/ Feedback welcome on this as well (e.g by creating issue here)

commented

@bwplotka we're just now starting to explore this space as mismatched versions of tooling is causing some pain points. Bingo looks interesting, but what prevents it from running on Windows? That would be a non-starter for us, unfortunately.

commented

We just don't use windows, so not yet tackled, that's it (: There are huge differences in how the path, shell (PowerShell!) and $PATH works, etc. Otherwise, there is no blocker TBH (: I added an issue on bingo to track this: bwplotka/bingo#26 it should be actually quite a quick fix to make it work if you guys want to help and contribute 💪

commented

I'm curious why tools.go with // +build tools? Wouldn't something like tools_test.go work as well?

commented

Hi Kamil :) #25922 (comment) To avoid init() side-effects

commented

@AlekSi Hi :) I was asking to use tools_test.go instead of tools.go. init() side-effects still apply but just for tests so 🤷 ... however I realized go test would pick it up and that would be much more annoying and not working:) Thanks for explanation.

commented

Change https://golang.org/cl/261499 mentions this issue: internal/tools: add a dummy package that imports mkwinsyscall

commented

In one of the projects we had a separate go.tools.mod file where we've defined our tooling dependencies.

Installation is simple but with 1 additional param, ex: go get -modfile=go.tools.mod github.com/foo/bar.

Example from open source https://github.com/mattermost/mattermost-server/blob/master/Makefile#L573 and https://github.com/mattermost/mattermost-server/blob/master/go.tools.mod (kindly ping @agnivade as one of the authors and Go contributor)

commented

Hey there, I'm not exactly sure what is it you wanted me to do?

commented

As mentioned above, @cristaloleg even separate go.tools.mod will not work for most of the cases.

I wrote about this in detail here but TL;DR on using go.tools.mod:

  • You don't fight with dependency conflicts between tools and your main project module, sure, but you still fight with dependency conflicts between multiple tools you pin in this go.tools.mod (unless you have one tool)
  • You cannot install automatically two tools with the same binary name.
  • You cannot pin two binaries from the same module under different version (go.mod does NOT support per package versioning)
  • In fact you cannot even pin just a single binary/tool for a huge module that has multiple of them.
  • If the tool you pin comes from the bigger module (e.g see prometheus mod) with multiple replace / exclude manual hackery (which is required for "broken" modules) - you are stuck. You can't pin such a tool or you need to manually fight a lot.
  • go.tools.mod requires go.mod to be in repo (!), so you cannot pin go tools in non-go projects.
  • Management is painful - you have to reference the tool via full path instead of just e.g tool name.

And there is more 🤗 but hope it's enough to motivate the bingo existence. Bingo solves all those painpoints. We recently released a new version v0.3.0 so you are more than welcome to try it out! I have a full-time job, and other much bigger open-source projects to maintain, so please use, contribute and help us to make this even better.

... and hopefully, this tool will inspire Go native tools for a similar user experience (happy to contribute anything!) 🤗

commented

re: @rsc in #25922 (comment)

Best practice remains creating the tools.go file.
It's true that go mod init does not auto-create a tools.go from dep's config,
but I think doing so is getting a bit beyond scope.

The best practice described currently triggers a go vet warning (see e.g. llir/llvm#196)

From llir/llvm/ir/enum/tools.go:

//go:build tools

package enum

import (
	_ "golang.org/x/tools/cmd/stringer"
)

Running go vet -tags tools github.com/llir/llvm/ir/enum results in the following warning:

ir/enum/tools.go:6:2: import "golang.org/x/tools/cmd/stringer" is a program, not an importable package

Is using a tools.go file with a go:build tools build tag still the recommended best practice for handling tools dependencies? Or is there another approach that is now recommended?

Cheers,
Robin

commented

@mewmew In this case, don't run go vet with -tags tools. The tools.go file is not meant to be part of a buildable package, so vet won't provide useful feedback on it. Without -tags tools, everything except go mod tidy will ignore tools.go.

commented

I've moved to installing tools.go dependencies using make with the power of go list:

install:
	go install $$(go list -f '{{join .Imports " "}}' tools.go)

I've not looked into this, but I feel //go:build tools should become obsolete doing so.

commented
commented

@rsc

Best practice remains creating the tools.go file. It's true that go mod init does not auto-create a tools.go from dep's config, but I think doing so is getting a bit beyond scope.

Another data point: I'm urgently porting an older large project from git submodules to go modules and tools.go approach breaks go list -deps, which I'm using as a precise audit step together with a custom GOPROXY with a more relaxed general allow list:

// +build tools

package x

import (
	_ "golang.org/x/tools/cmd/stringer"
)

C:\Users\egon\Desktop\x>go list -deps -tags tools -f "{{with .Module}}{{.}}{{end}}"
tools.go:6:2: import "golang.org/x/tools/cmd/stringer" is a program, not an importable package
golang.org/x/sys v0.0.0-20211019181941-9d821ace8654
golang.org/x/tools v0.1.8
...

I can get rid of the error by using -e, but I prefer hard errors. An alternative is to check the final binaries with go version -m or maybe complicate the GOPROXY infrastructure with more precise rules. Not a smooth experience...

@bcmills

commented

I think I'll just summarize my findings down here; it didn't all seem obvious and took some googling and tinkering, as some sources are a bit ambiguous on some of the points.

All native: tools.go

The best practice as endorsed by the Go team (#25922 (comment)).

//go:build tools
// +build tools

package yourpackage

import (
	_ "golang.org/x/tools/cmd/stringer"
	// ...
)

Change yourpackage to the name of the package that you put this in, e.g., use qux if you put it at the root of your module foo/bar/qux. Only use tools when you actually put it in a tools package (directory).

If you're on Go 1.17+, you don't need the // +build tools line.

go.mod

To add the dependencies and the latest versions of these tools to your go.mod and freshen the module cache, run

go mod tidy

Configure gopls

If you use gopls, configure it to include the tools build tag, e.g., for VSCode, add the following to your settings.json:

"gopls": {
  "build.buildFlags": ["-tags=tools"],
}

Install globally

You can install the go.mod versions of all the tools to your $GOBIN directory (probably not the best idea, expand for details) 
go install $(go list -f '{{join .Imports " "}}' tools.go)

Note that this can break things for you well outside the module that you run this in, so you're likely going to be better off installing specific versions of tools manually, at your discretion.

If you still feel like you want to have the tools built (they really do start faster that way), either of the following sections (on Makefile and bingo) might be right up your alley.

Makefile

Makefiles are very project-specific and everyone's setup is different, but I've found it useful to add this to my Makefile. 
toolsGo := tools.go
toolsDir := bin/tools
toolPkgs := $(shell go list -f '{{join .Imports " "}}' ${toolsGo})
toolCmds := $(foreach tool,$(notdir ${toolPkgs}),${toolsDir}/${tool})
$(foreach cmd,${toolCmds},$(eval $(notdir ${cmd})Cmd := ${cmd}))

go.mod: ${toolsGo}
	go mod tidy
	touch go.mod

${toolCmds}: go.mod
	go build -o $@ $(filter %/$(@F),${toolPkgs})

tools: ${toolCmds}
.PHONY: tools

If you decided to put your tools.go in a tools package, you would have to change the toolsGo variable to tools/tools.go. It may also make sense to flip the toolsDir variable to tools/bin. And maybe put this snippet in a tools/tools.mk, that you could include to reduce the clutter in your main Makefile.

You'd probably want to add the toolsDir directory to your .gitignore.

The magic $(foreach ... $(eval ... line defines <toolname>Cmd variables (e.g. stringerCmd, mockgenCmd, etc.) to be used in other recipes in your Makefile. For (a very rudimentary) example:

wire: ${wireCmd}
	${wireCmd} ./...
.PHONY: wire

Running make wire would now

  1. run go mod tidy to ensure go.mod is up to date with tools.go (if the latter is modified more recently than the former)
  2. build wire at the version that's defined in your go.mod (if it's not already built), and, finally,
  3. recursively generate all your provider and injector wiring.

You can also execute

make tools

and, if go.mod is older than tools.go, it will run go mod tidy, after which all the tools that you have defined in tools.go will get built under the toolsDir (as bin/tools/stringer, for example).

3rd party: bingo

You can use bingo to install version-suffixed executables of your module's tools in the global $GOBIN directory. This is an entirely different approach that avoids conflicts among tools and tool dependencies in larger projects, but it doesn't integrate with your module's go.mod (which might also be a good thing).

The author introduced this tool in #25922 (comment) and further expanded on it in #25922 (comment)

commented

I think golang needs to:

  1. Provide a way to define tool dependencies in go.mod only.
  2. Provide a way to run tool binaries, e.g. go execute <tool> <args>.

Edit: It appears go run package@version already provides point 2, so really only point 1 is missing for complete management of tool dependencies.

commented

@antichris, I still dont understand how I get the binary from that tools file and can use it in go generate?

I have this in the main package.

//go:build tools

package main

import (
	_ "github.com/golang-migrate/migrate/v4"
	_ "github.com/golang-migrate/migrate/v4/database/postgres"
	_ "github.com/golang-migrate/migrate/v4/source/github"
)

I ran go mod tidy but how can I use the binary now?

package main

//go:generate migrate -source=db
func main() {

}
$ go generate
main.go:3: running "migrate": exec: "migrate": executable file not found in $PATH
commented

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

commented

@bluebrown In general, the common practice would be running the tool using full package path in a go:generate directive, like so:

//go:generate go run golang.org/x/tools/cmd/stringer -type Foo

This would work if you have imported the respective package in your tools.go, e.g.:

import _ "golang.org/x/tools/cmd/stringer"

In your specific case, you'd have to import github.com/golang-migrate/migrate/v4/cmd/migrate:

import _ "github.com/golang-migrate/migrate/v4/cmd/migrate"

And, according to their documentation, add the appropriate build tags when running the COMMAND that you need:

//go:generate go run -tags postgres github.com/golang-migrate/migrate/v4/cmd/migrate -source=db COMMAND