tests fails under interactive shell (color issue)

Question

tests fails under interactive shell (color issue)

stacyharper opened this issue 2 years ago · comments

Hey there,

Looks like with my regular shell setup the tests fails. I have the issue with abuild (Alpine Linux) or with regular make test. I guess as an interactive shell, some git commands output are colored (a magenta "John Doe").

=== RUN   TestCommentAdd
Building identity cache... Done.
Building bug cache... Done.
    comment_add_test.go:32: 
        	Error Trace:	/home/willow/sources/git-bug/commands/comment_test.go:151
        	           				/home/willow/sources/git-bug/commands/comment_add_test.go:32
        	Error:      	Not equal: 
        	           	expected: "John Doe"
        	           	actual  : "\x1b[35mJohn Doe\x1b[0m"
        	           	
        	           	Diff:
        	           	--- Expected
        	           	+++ Actual
        	           	@@ -1 +1 @@
        	           	-John Doe
        	           	+John Doe
        	Test:       	TestCommentAdd
--- FAIL: TestCommentAdd (0.02s)
=== RUN   TestCommentEdit
Building identity cache... Done.
Building bug cache... Done.
    comment_edit_test.go:17: 
        	Error Trace:	/home/willow/sources/git-bug/commands/comment_edit_test.go:17
        	Error:      	Received unexpected error:
        	           	comment doesn't exist
        	Test:       	TestCommentEdit
--- FAIL: TestCommentEdit (0.03s)
=== RUN   TestComment
Building identity cache... Done.
Building bug cache... Done.
    comment_test.go:20: 
        	Error Trace:	/home/willow/sources/git-bug/commands/comment_test.go:151
        	           				/home/willow/sources/git-bug/commands/comment_test.go:20
        	Error:      	Not equal: 
        	           	expected: "John Doe"
        	           	actual  : "\x1b[35mJohn Doe\x1b[0m"
        	           	
        	           	Diff:
        	           	--- Expected
        	           	+++ Actual
        	           	@@ -1 +1 @@
        	           	-John Doe
        	           	+John Doe
        	Test:       	TestComment
--- FAIL: TestComment (0.02s)

I successfully built and test with piping "make test | less".

Michael Muré · Answer 1 · Mon Nov 21 2022 19:16:42 GMT+0800 (China Standard Time)

It looks like the tests somehow think that the output buffer is a terminal supporting colors. Quite odd indeed, in particular why that would happen to you and nowhere else.

It could be resolve easily by explicitly disabling color support, but it would be nice to understand what's happening.

Steve Moyer · Answer 2 · Mon Nov 21 2022 20:03:07 GMT+0800 (China Standard Time)

This is clearly my fault but it's a bit odd that tests pass without the -bench=. flag and fail if it's present. I'll have to remember to run tests via the Makefile every now and then (before pushing commits?). I'll admit I usually run tests using go test ./... or go test -v ./... just to make them faster.

My initial thought was that the shell used by the Makefile was subtly different but now I've got some investigation into the benchmarking environment. Piping the output to less works because that's disabling ANSI colors (you can see this with grep and many other *nix tools that colorize the output for humans).

Steve Moyer · Answer 3 · Sat Jan 07 2023 05:57:04 GMT+0800 (China Standard Time)

This is pretty bizarre - when the tests are run with benchmarking (go test -v -bench=. ./...), os.Stdout contains an FD that is in fact a TTY. When the tests are run without benchmarking, os.Stdout contains an FD that is NOT a TTY. Here's the package/method/function stack I've investigated so far - the problem is in Go cmd runner:

In fatih/color, NoColor is false during benchmarking and true during normal test runs (https://github.com/fatih/color/blob/a05da93ebe62ca9fc6791d3376ec4dad01196448/color.go#L22-L23)
In mattn/go-isatty, IsTerminal(os.Stdout.Fd() is true during benchmarking and false during normal test runs (https://github.com/mattn/go-isatty/blob/ed75e619dc0f0489fd4062163a7d061eaa249b9c/isatty_tcgets.go#L10-L11)
In x/sys/unix, unix.IoctlGetTermios(int(os.Stdout.Fd()), unix.TCGETS) returns a Termios during benchmarking and an error during normal testing (https://cs.opensource.google/go/x/sys/+/refs/tags/v0.4.0:unix/ioctl.go;drc=b36040661688bbbde06c38ea80af105ebf21415b;l=71)
In command/go/internal/test, output is sent to Stdout (a TTY) if a) there's not an explicit test package, b) benchmarking is enabled, c) if fuzzing is enabled or d) if only one package is explicitly requested. Otherwise, the test output is buffered and Stdout is NOT a TTY (so the ANSI escape sequences aren't generated).

Failing cases:

go test -v -bench=. ./...
go test -v -fuzz=. ./...
cd commands/bug && go test -v ./...

Passing case:

go test -v ./...

I've asked for advice in the #testing channel of the Gopher's Slack - my opinion is that buffered or not, the output shouldn't change in this case but I'm waiting for advice on either a work-around or on submitting an issue with a proper description. At this point, I've got a tiny and concise program demonstrates this error so the good news is that it's not a problem with git-bug.

Steve Moyer · Answer 4 · Sat Jan 07 2023 22:57:17 GMT+0800 (China Standard Time)

Submitted to the Go issue tracker - golang/go#57671

Steve Moyer · Answer 5 · Sun Jan 08 2023 00:04:29 GMT+0800 (China Standard Time)

It turns out this issue was discussed and rejected in golang/go#34877. Fortunately, I already had a fix ready and have opened the associated PR - #984.