Giters

goproxy / goproxy

A minimalist Go module proxy handler.

Home Page:https://pkg.go.dev/github.com/goproxy/goproxy

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] go get stuck and server output git error

adlternative opened this issue · comments

commented

I have set up a server on localhost:9080 using goproxy, but it gets stuck when I try to execute go get.

client:

GOPROXY=http://localhost:9080 go get gitlab.xxx.com/aaa/bbb@latest
go: downloading gitlab.xxx.com/aaa/bbb v0.0.0-20230815123511-328a75f65a51

server log:

2023/08/31 19:01:53 goproxy.go:391: goproxy: failed to list module version: gitlab.xxx.com/@v/list: module xxx.com: git ls-remote -q origin in /Users/adl/go/pkg/mod/cache/vcs/73e7c5efccc154f70f696b7ea0cb7a2a8e4f7960832a7bf61d0232685fc3163e: exit status 128:
	fatal: no path specified; see 'git help pull' for valid url syntax
2023/08/31 19:01:54 goproxy.go:391: goproxy: failed to list module version: gitlab.xxx.com/bbb/@v/list: module gitlab.xxx/bbb: git ls-remote -q origin in /Users/adl/go/pkg/mod/cache/vcs/f56dcfdca177b425cf074e4d7907ec28d80ec30aee1b92f4f93290aa4b447dec: exit status 128:
	fatal: protocol error: bad line length character: repo

server code:

	http.ListenAndServe(":9080", &goproxy.Goproxy{
		GoBinEnv: append(
			os.Environ(),
			"GOPROXY=https://goproxy.cn",  // 使用 Goproxy.cn 作为上游代理
			"GOPRIVATE=*.xxx.com", // 解决私有模块的拉取问题（比如可以配置成公司内部的代码源）
		),
		ProxiedSUMDBs: []string{
			"sum.golang.org https://goproxy.cn/sumdb/sum.golang.org", // 代理默认的校验和数据库
		},
		ErrorLogger: logger,
	})

ENV:

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/adl/Library/Caches/go-build"
GOENV="/Users/adl/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/adl/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/adl/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bj/t_584mj969b8zchzmqh_l4xh0000gp/T/go-build3868987923=/tmp/go-build -gno-record-gcc-switches -fno-common"

$ git version
git version 2.39.2 (Apple Git-143)
commented

Sorry for the late response, I missed the email.

server log:

2023/08/31 19:01:53 goproxy.go:391: goproxy: failed to list module version: gitlab.xxx.com/@v/list: module xxx.com: git ls-remote -q origin in /Users/adl/go/pkg/mod/cache/vcs/73e7c5efccc154f70f696b7ea0cb7a2a8e4f7960832a7bf61d0232685fc3163e: exit status 128:
	fatal: no path specified; see 'git help pull' for valid url syntax
2023/08/31 19:01:54 goproxy.go:391: goproxy: failed to list module version: gitlab.xxx.com/bbb/@v/list: module gitlab.xxx/bbb: git ls-remote -q origin in /Users/adl/go/pkg/mod/cache/vcs/f56dcfdca177b425cf074e4d7907ec28d80ec30aee1b92f4f93290aa4b447dec: exit status 128:
	fatal: protocol error: bad line length character: repo

Please be aware that these two lines of server logs you provided are expected. It's just how Go modules work. See #6 (comment) for details.

As for you mentioning that you're stuck on go get, my guess is that there's a problem with your server side network. I'm not sure, I'd need more details.

commented

I'm not 100% sure it's related and I don't have a reproducible case I can share but here's what is happening.

Context:

  • We're using a private Github repo, goproxy accesses using GOPROXY=direct
  • Goproxy is used as a library (just goproxy.Goproxy{})
  • Only requests for private dependencies go through goproxy
  • I use goenv to manage Golang versions
  • It happens locally and in CI/CD

Bug:

  • It hangs during go mod tidy (when ran from scratch, after go clean -modcache)
  • It's stuck running commands using os/exec

Conditions:

  • It only happens when goproxy is ran usinggo run; it doesn't happen for a compiled version
  • It only when both goproxy and go mod tidy use the same Golang version (I tried 1.19.8 and 1.19.7)
  • It doesn't happen when they use different Golang versions

The behavior is consistent. The relevant stack trace:

goroutine 68 [runnable]:
syscall.syscall6(0x100d785?, 0x10c7a8c?, 0xc000140ae0?, 0x2c?, 0x100c0004e46c0?, 0x1000000000003?, 0x18ae548?)
	/Users/martinb/.goenv/versions/1.21.4/src/runtime/sys_darwin.go:45 +0x98 fp=0xc0004e45e8 sp=0xc0004e4528 pc=0x1063ef8
syscall.wait4(0xc0004e4670?, 0x100db05?, 0x90?, 0x13f60c0?)
	/Users/martinb/.goenv/versions/1.21.4/src/syscall/zsyscall_darwin_amd64.go:43 +0x45 fp=0xc0004e4648 sp=0xc0004e45e8 pc=0x1080325
syscall.Wait4(0xc000127210?, 0xc0004e46a4, 0x0?, 0x0?)
	/Users/martinb/.goenv/versions/1.21.4/src/syscall/syscall_bsd.go:144 +0x25 fp=0xc0004e4680 sp=0xc0004e4648 pc=0x107f065
os.(*Process).wait(0xc000140b70)
	/Users/martinb/.goenv/versions/1.21.4/src/os/exec_unix.go:43 +0x6d fp=0xc0004e46d8 sp=0xc0004e4680 pc=0x10c392d
os.(*Process).Wait(...)
	/Users/martinb/.goenv/versions/1.21.4/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000134420)
	/Users/martinb/.goenv/versions/1.21.4/src/os/exec/exec.go:890 +0x45 fp=0xc0004e4740 sp=0xc0004e46d8 pc=0x128e9c5
os/exec.(*Cmd).Run(0xc0004e47a0?)
	/Users/martinb/.goenv/versions/1.21.4/src/os/exec/exec.go:590 +0x2d fp=0xc0004e4760 sp=0xc0004e4740 pc=0x128d5ad
os/exec.(*Cmd).Output(0xc000134420)
	/Users/martinb/.goenv/versions/1.21.4/src/os/exec/exec.go:984 +0xb6 fp=0xc0004e47b0 sp=0xc0004e4760 pc=0x128f096
github.com/goproxy/goproxy.(*fetch).doDirect(0xc0004c2680, {0x14a6e50, 0x173bc00})
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/fetch.go:229 +0x3a5 fp=0xc0004e49d0 sp=0xc0004e47b0 pc=0x12c3265
github.com/goproxy/goproxy.(*fetch).do.func2()
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/fetch.go:121 +0x28 fp=0xc0004e4a00 sp=0xc0004e49d0 pc=0x12c2488
github.com/goproxy/goproxy.walkGOPROXY({0x1414014?, 0xffffffffffffffff?}, 0xc0004e4ac0, 0xc0004e4a98, 0x14411b8)
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:561 +0x1e5 fp=0xc0004e4a68 sp=0xc0004e4a00 pc=0x12c8de5
github.com/goproxy/goproxy.(*fetch).do(0xc0004c2680, {0x14a6e50, 0x173bc00})
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/fetch.go:115 +0x114 fp=0xc0004e4af8 sp=0xc0004e4a68 pc=0x12c23d4
github.com/goproxy/goproxy.(*Goproxy).serveFetchDownload(0x0?, {0x14a6430, 0xc00049a4c0}, 0xc0004e5148, 0xc0004c2680)
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:374 +0x8a fp=0xc0004e4c70 sp=0xc0004e4af8 pc=0x12c722a
github.com/goproxy/goproxy.(*Goproxy).serveFetch.func2()
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:337 +0x27 fp=0xc0004e4ca8 sp=0xc0004e4c70 pc=0x12c7127
github.com/goproxy/goproxy.(*Goproxy).serveCache(0x0?, {0x14a6430, 0xc00049a4c0}, 0xc0004e5148, {0xc0004b4af5, 0x3e}, {0x140b415, 0xf}, 0xc0004e4e18?, 0xc0004e4ed0)
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:484 +0x1bc fp=0xc0004e4d68 sp=0xc0004e4ca8 pc=0x12c859c
github.com/goproxy/goproxy.(*Goproxy).serveFetch(0xc0000e0f00, {0x14a6430, 0xc00049a4c0}, 0xc0004e5148, {0xc0004b4af5?, 0x30?}, {0xc0004a0b40?, 0x100d785?})
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:336 +0x819 fp=0xc0004e4f50 sp=0xc0004e4d68 pc=0x12c6e39
github.com/goproxy/goproxy.(*Goproxy).ServeHTTP(0xc0000e0f00, {0x14a6430, 0xc00049a4c0}, 0xc0004e5148)
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:304 +0x3e8 fp=0xc0004e5020 sp=0xc0004e4f50 pc=0x12c6508
main.(*LocalProjects).fetch(0xc000081920, {0xc0004c94b8?, 0xc0004ba174?}, {0xc0004ba001, 0x3e})
	/Users/martinb/dev/Tooploox/DTS/conrad/experiments/goproxy/main.go:115 +0x22a fp=0xc0004e5250 sp=0xc0004e5020 pc=0x13435aa
main.(*LocalProjects).Get(0xc000081920, {0x14a6da8, 0xc0004820a0}, {0xc0004ba001, 0x3e})
	/Users/martinb/dev/Tooploox/DTS/conrad/experiments/goproxy/main.go:271 +0x7f3 fp=0xc0004e5798 sp=0xc0004e5250 pc=0x13458b3
github.com/goproxy/goproxy.(*Goproxy).cache(0xc000026d80?, {0x14a6da8?, 0xc0004820a0?}, {0xc0004ba001?, 0xc0004ba03b?})
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:500 +0x30 fp=0xc0004e57d0 sp=0xc0004e5798 pc=0x12c8750
github.com/goproxy/goproxy.(*Goproxy).serveCache(0x0?, {0x14a65b0, 0xc0004be000}, 0xc00043a000, {0xc0004ba001, 0x3e}, {0x140b415, 0xf}, 0xc000380940?, 0xc0004e59f8)
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:481 +0xac fp=0xc0004e5890 sp=0xc0004e57d0 pc=0x12c848c
github.com/goproxy/goproxy.(*Goproxy).serveFetch(0xc0000e1040, {0x14a65b0, 0xc0004be000}, 0xc00043a000, {0xc0004ba001?, 0x10?}, {0xc0004a02d0?, 0x13ab7c0?})
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:336 +0x819 fp=0xc0004e5a78 sp=0xc0004e5890 pc=0x12c6e39
github.com/goproxy/goproxy.(*Goproxy).ServeHTTP(0xc0000e1040, {0x14a65b0, 0xc0004be000}, 0xc00043a000)
	/Users/martinb/go/1.21.4/pkg/mod/github.com/goproxy/goproxy@v0.15.1/goproxy.go:304 +0x3e8 fp=0xc0004e5b48 sp=0xc0004e5a78 pc=0x12c6508
net/http.serverHandler.ServeHTTP({0xc0004200f0?}, {0x14a65b0?, 0xc0004be000?}, 0x6?)
	/Users/martinb/.goenv/versions/1.21.4/src/net/http/server.go:2938 +0x8e fp=0xc0004e5b78 sp=0xc0004e5b48 pc=0x126abce
net/http.(*conn).serve(0xc000434000, {0x14a6d70, 0xc0001ac2a0})
	/Users/martinb/.goenv/versions/1.21.4/src/net/http/server.go:2009 +0x5f4 fp=0xc0004e5fb8 sp=0xc0004e5b78 pc=0x1267b14
net/http.(*Server).Serve.func3()
	/Users/martinb/.goenv/versions/1.21.4/src/net/http/server.go:3086 +0x28 fp=0xc0004e5fe0 sp=0xc0004e5fb8 pc=0x126b3e8
runtime.goexit()
	/Users/martinb/.goenv/versions/1.21.4/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e5fe8 sp=0xc0004e5fe0 pc=0x10673c1
created by net/http.(*Server).Serve in goroutine 1
	/Users/martinb/.goenv/versions/1.21.4/src/net/http/server.go:3086 +0x5cb
commented

Hi @bilus, thanks for the great analysis!

Are your Goproxy process and go mod tidy running in the same environment? If so, try running them with different GOMODCACHE.

For example:

GOMODCACHE=/tmp/goproxy-gomodcache go run goproxy.go

and

GOMODCACHE=/tmp/gomodtidy-gomodcache go mod tidy

It seems there might be a deadlock somewhere. Might be something to do with go run's process tree, I'll look into it later.

commented

Yup, it's caused by a deadlock in GOMODCACHE, and it's 100% reproducible.

Start a Goproxy server:

GOPROXY=direct GOMODCACHE=/tmp/goproxy-gomodcache \
  go run github.com/goproxy/goproxy/cmd/goproxy@latest \
  server --address localhost:8080 --cacher-dir /tmp/goproxy-caches

Fetch something:

GOPROXY=http://localhost:8080 GOMODCACHE=/tmp/goproxy-gomodcache \
  go mod download -json golang.org/x/text@latest

Then it gets stuck.

Run lsof | grep "/tmp/goproxy-gomodcache", you should be able to see two processes holding the same lock file:

... /tmp/goproxy-gomodcache/cache/download/golang.org/x/text/@v/v0.14.0.lock
... /tmp/goproxy-gomodcache/cache/download/golang.org/x/text/@v/v0.14.0.lock

While this is true, I don't consider it a bug; it's more like an edge case. This is because, in most cases, Goproxy should be run in a dedicated environment, such as a container. More importantly, this locking mechanism prevents duplicate VCS cloning operations, which is actually quite beneficial.

However, it definitely needs be mentioned in the docs that Goproxy should always have its own GOMODCACHE.

commented

Thanks for a super quick response, I appreciate it.

Yes, it should be mentioned in the documentation, it cost me a lot of time to figure it out. :>

Suggestions: You could easily time out and report & log an error. Plus there's too little logging available, please consider adding an option to turn on debugging logs (including setting -x flag to go mod). I had to patch my copy to figure out what's going on.

commented

Additional flags can be set via GOFLAGS. For example, GOFLAGS=-modcacherw works great for me.

Enabling verbose logging for go command calls can be a bit tricky since we can't simply redirect exec.Cmd.Stderr to os.Stderr. I'll need some time to figure out the best approach for achieving this.