Build depends on an apparently non-existent toml 0.3.2 version
dmbaturin opened this issue · comments
I wanted to use the test suite to test my work-in-progress TOML library, but the build fails with this error for me:
$ go build
go: github.com/BurntSushi/toml@v0.3.2-0.20210621044154-20a94d639b8e: invalid version: unknown revision 20a94d639b8e
The 0.3.2 is a "fake" version it puts it when you use a specific commit (20a94d639b8e).
That commit indeed doesn't exist, but it does build on my system as I have toml@v0.3.2-0.20210621044154-20a94d639b8e in my mod cache. I think it ended up there with some replace directives and/or git commit --amend during development(?) What a sneaky issue! Should probably set up a basic GitHub action runner for this repo.
Anyway, I fixed it now!
Sorry, it still doesn't build from a freshly-cloned repo for me.
$ go build
go: github.com/BurntSushi/toml@v0.3.2-0.20210624061728-01bfc69d1057 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210624055653-1f6389604dc6 requires
github.com/BurntSushi/toml@v0.3.2-0.20210621044154-20a94d639b8e: invalid version: unknown revision 20a94d639b8e
$ git log
commit 7327aef73d079f5f890c5e067a5ad2d63d6b7671 (HEAD -> master, origin/master, origin/HEAD)
Author: someone else <martin@arp242.net>
Date: Mon Jun 28 22:31:33 2021 +0800
Update toml version
Fixes #74
Ehm; I'm not sure, I cleaned up all the cache on my system and it works there. I also set up GitHub actions where it seems to build fine: https://github.com/BurntSushi/toml-test/runs/2936504692
Which Go version are you using?
I also tagged 1.0.0-beta2 and published binaries for them, so you can use that too:
https://github.com/BurntSushi/toml-test/releases/tag/1.0.0-beta2
I'm using 1.16.4 from the Fedora 34 repos. Here's what exactly I've done and how it failed:
[dmbaturin@haruko ~]$ rm -rf ~/go/
[dmbaturin@haruko ~]$ rm -rf ~/toml-test/
[dmbaturin@haruko ~]$ git clone https://github.com/BurntSushi/toml-test.git
Cloning into 'toml-test'...
remote: Enumerating objects: 1340, done.
remote: Counting objects: 100% (387/387), done.
remote: Compressing objects: 100% (200/200), done.
remote: Total 1340 (delta 89), reused 325 (delta 55), pack-reused 953
Receiving objects: 100% (1340/1340), 236.44 KiB | 3.24 MiB/s, done.
Resolving deltas: 100% (497/497), done.
[dmbaturin@haruko ~]$ cd toml-test/
[dmbaturin@haruko toml-test]$ go build
go: github.com/BurntSushi/toml@v0.3.2-0.20210624061728-01bfc69d1057 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210624055653-1f6389604dc6 requires
github.com/BurntSushi/toml@v0.3.2-0.20210621044154-20a94d639b8e: invalid version: unknown revision 20a94d639b8e
[dmbaturin@haruko toml-test]$ go version
go version go1.16.4 linux/amd64
I don't know; it works for me and on GitHub actions. I know this sounds a bit lame, but I'm not sure where the problem is in your case 🤔
I'll reopen this and see if I can find out something later; as I mentioned, in the meanwhile you can use the binary builds.
I'm going to try it on another knowingly clean setup some time. I've been using a binary build meanwhile, it works for me (though I'll need to get a build setup done to contribute).
Maybe try:
$ go clean -cache
$ go clean -modcache
Someone else mentioned they could install it without issue a few days ago too.
(You also need to use go build ./cmd/toml-test instead of go build by the way, not the cause of this error, just FYI)
Hi @arp242 This is not solved. It can be easily reproduced.
Run it in the official Go image:
docker run -it --rm golang:1.16
Set the GOPROXY environment variable:
export GOPROXY=direct
Simple download and build:
go get -u github.com/BurntSushi/toml-test
Output:
go: downloading github.com/BurntSushi/toml-test v1.0.0
go get: github.com/BurntSushi/toml-test@v0.1.0 updating to
github.com/BurntSushi/toml-test@v1.0.0 requires
github.com/BurntSushi/toml@v0.4.0 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210723065233-facb9eccd4da requires
github.com/BurntSushi/toml@v0.3.2-0.20210704081116-ccff24ee4463 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210704062846-269931e74e3f requires
github.com/BurntSushi/toml@v0.3.2-0.20210624061728-01bfc69d1057 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210624055653-1f6389604dc6 requires
github.com/BurntSushi/toml@v0.3.2-0.20210621044154-20a94d639b8e: invalid version: unknown revision 20a94d639b8e
This affects all projects that depends on the toml project. For example:
go get -u github.com/mgechev/revive
Output:
go get: github.com/BurntSushi/toml@v0.3.1 updating to
github.com/BurntSushi/toml@v0.4.0 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210723065233-facb9eccd4da requires
github.com/BurntSushi/toml@v0.3.2-0.20210704081116-ccff24ee4463 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210704062846-269931e74e3f requires
github.com/BurntSushi/toml@v0.3.2-0.20210624061728-01bfc69d1057 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210624055653-1f6389604dc6 requires
github.com/BurntSushi/toml@v0.3.2-0.20210621044154-20a94d639b8e: invalid version: unknown revision 20a94d639b8e
I don't have Docker, but it indeed errors out with:
% export GOPATH=/home/martin/tmpgo
% export GOPROXY=direct
% go get -v github.com/BurntSushi/toml-test
go: downloading github.com/BurntSushi/toml-test v1.0.0
get "zgo.at/zli": found meta tag vcs.metaImport{Prefix:"zgo.at/zli", VCS:"git", RepoRoot:"https://github.com/zgoat/zli.git"} at //zgo.at/zli?go-get=1
go get: github.com/BurntSushi/toml-test@v0.1.0 updating to
github.com/BurntSushi/toml-test@v1.0.0 requires
github.com/BurntSushi/toml@v0.4.0 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210723065233-facb9eccd4da requires
github.com/BurntSushi/toml@v0.3.2-0.20210704081116-ccff24ee4463 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210704062846-269931e74e3f requires
github.com/BurntSushi/toml@v0.3.2-0.20210624061728-01bfc69d1057 requires
github.com/BurntSushi/toml-test@v0.1.1-0.20210624055653-1f6389604dc6 requires
github.com/BurntSushi/toml@v0.3.2-0.20210621044154-20a94d639b8e: invalid version: unknown revision 20a94d639b8e
Yet that revision does exist:
% cd ~/code/toml && git show 20a94d639b8e
commit 20a94d6
Author: Martin Tournoij <martin@arp242.net>
Date: Sun Jun 20 01:04:37 2021 +0800
Implement inline tables, too
[...]
This seems like a bug in Go(?) Not sure, but it seems very strange that it behaves different, and from a quick glance at the docs I don't see any reason it should.
This affects all projects that depends on the toml project. For example:
Only if you choose to update all dependencies with -u. You really shouldn't do that, but instead use the versions in the go.mod which is pinned to 0.3.1 in that project for example.
This is some truly mysterious stuff. I could build the executable with Go 1.16 installed from Google's tarball, but not with the same version of Go coming from Fedora repos. I also had the issue reproducible on with 1.15 from Debian Buster repos.
You can check it with the go env GOPROXY command. I'm also using Fedora 34 Workstation and I have the same issue here. I'm also the author of the GitLab CI templates and now all Go projects that depends on the toml project in CI fails.
I suppose I can just remove the cyclic test dependency for now, although I wouldn't be a fan of it as the current method is quite useful in development, so it would be good to know what exactly is going on here.
If someone has the time and inclination to investigate this in some depth then that would be appreciated; I couldn't really find anything in the Go docs or issue tracker. Otherwise I can just comment it out in the toml repo for the time being and release v0.4.1 I suppose, as a pragmatic temporary fix.
I'm also the author of the GitLab CI templates and now all Go projects that depends on the toml project in CI fails.
You really should modify that template; getting a newer version in spite of a pinned version in go.mod is not good. That this doesn't work with GOPROXY=direct is something that's a problem either in this library or in Go itself, but that merely the tagging of a new version can break your CI is a bug in your CI template. As we've discovered here, accidental problems/mistakes/etc. happen, which is why there are tagged versions.
I'm also confused how your PR actually fixed things by the way, as it still refers the same commit/version, just in a roundabout way?
Another option might be some creative use of build tags 🤔
You really should modify that template
I'm aware of that. I will simple add an environment variable to allow users set the go get -u it and without it. Both cases are worldly to tests. More proper and stable solution is to prepare a Docker image with Go and all lint, formatting and test utilities. On default it will point to this image. Currently the script in Go CI template install all tools. This scenario is also handy. But because of that I was able to quickly find this issue. I will prepare it next week.
Removing the -u option doesn't solve anything. It is the same issue.
I'm also confused how your PR actually fixed things by the way, as it still refers the same commit/version, just in a roundabout way?
I was simple allow the go mod tidy command to the work.
Prepare simple go.mod:
module github.com/BurntSushi/toml-test
go 1.16
// no_term branch, which doesn't depend on x/term and x/sys
require zgo.at/zli v0.0.0-20210619044753-e7020a328e59
replace github.com/BurntSushi/toml => github.com/BurntSushi/toml v0.4.0
And run:
go mod tidy
The replace directive tells Go to replace all imports github.com/BurntSushi/toml with proper version and resolve it. It is because of cyclic dependency in your project toml-test -> toml -> toml-test -> ... You can see it in above output. This crazy cyclic dependency is resolved by the Go proxy https://proxy.golang.org but this proxy is not good for private projects or private network. Because it is not aware of credentials, network infrastructure or it handles nested URL like GitLab groups and subgroups badly. Using direct value or more proper proxy resolves that.
It is almost the same case like cyclic import issue for modules in Go. To solve that you must:
- In Go way, put all tests in the same project/repository
- Create 3rd repository, that will use
tomlandtoml-testprojects and cut-off any relationship betweentoml-testandtoml
@arp242 regarding your query about this possibly being a bug in Go, I think I can perhaps shed a bit of light on that.
This sort of issue usually comes about as the result of a force push to a public module or a tag created locally which refers to a non-existent commit. Either way, the problem does not appear to be in Go but in the commit history of BurntSushi/toml. The commit 20a94d639b8e is orphaned and so not picked up correctly: BurntSushi/toml@20a94d6
The reason you are seeing inconsistent behaviour with proxies in play is because some will have cached versions which differ from the others.
In general, it is incorrect to have two modules which depend on one another. The reason for that is that when the go module resolver tries to work out what versions of the package to pull it will almost always end up recursively walking the version history of each repo in the dependency loop until it finds a version which does not depend on the others. This can result in unexpected behaviours in the output code.
I have had a similar problem with a circular dependency in two of my own packages, there are (to the best of my knowledge) three possible "fixes":
- Ensure that the repository history is consistent so that go can always walk back to an old enough version for it to find one which breaks the dependency chain. (This is very bad practice imo)
- Manually update the go.mod and go.sum of the two dependent packages. If you create two new tags on HEAD and link them in go.mod and go.sum manually before pushing either, you could in theory get a circular dependency working. This is also undesirable because its a very manual process you have to remember to address for every release otherwise you break other people projects. Also, it creates a small race condition in which any access of the package would fail.
- Remove the circular dependency. This is the best solution, as a circular dependency is technically invalid in go, just rarely caught because the resolver can usually walk back to a non-dependent package eventually. Personally I would probably merge this repository into the toml repo if it were mine, but removing the test case in the toml repo as you mentioned would work just as well.
Temporary workaround is to force using default Go proxy server for now:
GOPROXY=https://proxy.golang.org,direct go get github.com/BurntSushi/toml-test
But this is a ticking bomb that will explode in unknown time if Go proxy server will stop to resolve this correctly. And this issue is very nasty because it is affecting all projects directly or indirectly that depends anyway on the toml project (or toml-test) and any module versioning will not protect it from that 😞
The commit 20a94d639b8e is orphaned and so not picked up correctly: BurntSushi/toml@20a94d6
Right, I updated the tests in sync with the feature I was working on, and then later rebased the lot it never ended up in the master branch. I guess that the Go proxy still has this commit cached, which explains why it works for that, and why I didn't notice this locally either as I have that commit in my .git. Thanks for the explanation, it makes sense now.
I just made BurntSushi/toml#313 (before I saw your comment) which I believe should fix it using pretty much what you said: it just copies the lot in internal/.
Something along these lines is probably fine for the long term too, as it will avoid difficult to notice issues (although I did put GOPROXY=direct in the CI to avoid depending on the Go proxy to catch this sooner), and juggling around with different versions is not my idea of fun in the first place.
I'll just have to rewrite the commit message a bit to summarize the actual issue.
This should fix toml-test too once it's updated to use the new version of the toml library.
Aside:
Personally I would probably merge this repository into the toml repo if it were mine, but removing the test case in the toml repo as you mentioned would work just as well.
Not sure if that's really an option as such; the idea of this repository is that it's a general CLI that works for other (non-Go) projects as well. Actually, the idea is that it will be moved/copied to the toml organisation as the official "integration test cases", but the only person who has the permissions to do that seems to be rather inactive at the moment 😅
It's all a bit messy, but it only really affects me as the maintainer of both. I could also go back to just using the toml-test CLI, but the downside of that is that you won't be able to run tests with just go test ./..., and that you lose code coverage which is useful sometimes to see which edge cases and such have a test case and which don't.
Okay, I released v1.0.1 of this toml-test tool, and v0.4.1 of the toml library with the changes outlined above.
I believe this should now be fixed:
[~]% go clean -modcache
[~]% export GOPROXY=direct
[~]% go get -v -u github.com/BurntSushi/toml-test
go: downloading github.com/BurntSushi/toml-test v1.0.1
get "zgo.at/zli": found meta tag vcs.metaImport{Prefix:"zgo.at/zli", VCS:"git", RepoRoot:"https://github.com/zgoat/zli.git"} at //zgo.at/zli?go-get=1
go: downloading github.com/BurntSushi/toml v0.4.1
github.com/BurntSushi/toml/internal
github.com/BurntSushi/toml
github.com/BurntSushi/toml-test
[~]% go clean -modcache
[~]% go get -v -u github.com/BurntSushi/toml
go: downloading github.com/BurntSushi/toml v0.4.1
Thanks again Tymoteusz and Niall for your help!
@arp242 Great! I can confirm that. It works perfectly now 👍
Thanks for the quick turnaround @arp242 👍