TearDownTest is being executed while parallel sub tests are still running

Question

TearDownTest is being executed while parallel sub tests are still running

swithek opened this issue 6 years ago · comments

Consider the following example:

type Suite struct {
	suite.Suite
}

func (s *Suite) TearDownSuite() {
	s.T().Log(">>>>>> suite tear down")
}

func (s *Suite) TearDownTest() {
	s.T().Log(">>>>>> single test tear down")
}

func (s *Suite) TestOne() {
	for _, v := range []string{"sub1", "sub2", "sub3"} {
		s.Run(v, func() {
			s.T().Parallel()
		})
	}
}

func (s *Suite) TestTwo() {
	for _, v := range []string{"sub1", "sub2", "sub3"} {
		s.Run(v, func() {
			s.T().Parallel()
		})
	}
}

func TestLogic(t *testing.T) {
	suite.Run(t, &Suite{})
}

After executing that you would get an output similar to this:

=== RUN   TestLogic
=== RUN   TestLogic/TestOne
=== RUN   TestLogic/TestOne/sub1
=== PAUSE TestLogic/TestOne/sub1
=== RUN   TestLogic/TestOne/sub2
=== PAUSE TestLogic/TestOne/sub2
=== RUN   TestLogic/TestOne/sub3
=== PAUSE TestLogic/TestOne/sub3
    TestLogic/TestOne: prog.go:18: >>>>>> single test tear down
=== CONT  TestLogic/TestOne/sub1
=== CONT  TestLogic/TestOne/sub3
=== CONT  TestLogic/TestOne/sub2 >>> test finished here
=== RUN   TestLogic/TestTwo
=== RUN   TestLogic/TestTwo/sub1
=== PAUSE TestLogic/TestTwo/sub1
=== RUN   TestLogic/TestTwo/sub2
=== PAUSE TestLogic/TestTwo/sub2
=== RUN   TestLogic/TestTwo/sub3
=== PAUSE TestLogic/TestTwo/sub3
    TestLogic/TestTwo: prog.go:18: >>>>>> single test tear down
=== CONT  TestLogic/TestTwo/sub1
=== CONT  TestLogic/TestTwo/sub3
=== CONT  TestLogic/TestTwo/sub2 >>> test finished here
    TestLogic: prog.go:14: >>>>>> suite tear down

Live example: https://play.golang.com/p/fPY3DZFNNok

It is clear that TearDownTest is being executed before all parallel sub tests finish. I see that a similar issue was addressed in #799 and #466 but not resolved in its entirety.

The version in use is 1.5.1

Boyan Soubachov · Answer 1 · Sun May 03 2020 18:09:55 GMT+0800 (China Standard Time)

Do you mind running a test and confirming whether this happens with version 1.4.0?

swithek · Answer 2 · Mon May 04 2020 01:41:45 GMT+0800 (China Standard Time)

I've just tried running this test suite on 1.4.0 and the result was the same.

Boyan Soubachov · Answer 3 · Mon May 04 2020 19:06:13 GMT+0800 (China Standard Time)

Thank you, I will have a look.

Astrid Howard · Answer 4 · Tue Jun 22 2021 05:28:35 GMT+0800 (China Standard Time)

This occurs for me with 1.6.1 as well.

Aniruddha Maru · Answer 5 · Fri Jul 30 2021 03:29:04 GMT+0800 (China Standard Time)

It seems like this could be solved by just using t.Cleanup for both tearDownSuite and tearDownTest. Is there an obvious reason for not using cleanup? If not, I can put up a PR for this change.

Bracken · Answer 6 · Fri Jul 30 2021 03:44:53 GMT+0800 (China Standard Time)

t.Cleanup is a Go 1.14 feature, this would need us to update the go.mod to 1.14 and drop support of go 1.13. Not necessarily a bad thing, but needs to be done with intent. Is there an approach that works in 1.13?

Aniruddha Maru · Answer 7 · Fri Jul 30 2021 05:26:57 GMT+0800 (China Standard Time)

t.Cleanup is a Go 1.14 feature, this would need us to update the go.mod to 1.14 and drop support of go 1.13. Not necessarily a bad thing, but needs to be done with intent. Is there an approach that works in 1.13?

oh I see, well, worst case, we can do a conditional fix for 1.14+ but let me dig into 1.13 code.

Aniruddha Maru · Answer 8 · Fri Jul 30 2021 07:35:13 GMT+0800 (China Standard Time)

@brackendawson looks like the only option for <=1.13 is to use t.Run to create a new group within a suite and call all the tests within that group. This works because t.Run will not return until all sub-tests have finished which gives us a clean way to run teardown. The only downside I can think of is one additional level of nesting in tests, which mostly just affects output? tbf, I think all values within a suite must be run in a group anyway, since that aligns with stdlib's expectations, but perhaps there's a reason to not do this?

Aniruddha Maru · Answer 9 · Fri Jul 30 2021 09:06:04 GMT+0800 (China Standard Time)

@brackendawson although, Suite.SetT also got me thinking - is Suite even concurrent-safe? Because it is being run from multiple routines here, and if there are multiple parallel tests within a suite, they will overwrite each other's T object.

Bracken · Answer 10 · Sat Jul 31 2021 01:10:46 GMT+0800 (China Standard Time)

So on the original bug, I'm thinking we could:

Stop using testing.InternalTest here and instead define that type privately as suite.internalTest.
Give suite.internalTest a new field done of type chan struct{}.
Defer a close of that channel at the top the function we use in suite.internalTest.F (F will need to close over the channel where it is defined outside of the struct literal).
After here range over the tests again and this time just read one value from their done channels.

That should stop us from calling teardowns early, do you agree?

The Suite.SetT() call inside F you point out is definitely not safe if the user calls s.T().Parallel() in their test. Fortunately Suite.t is unexported and guarded behind Suite.T(). Unfortunately all the tests are the same anonymous function so we genuinely have no way to tell which test called suite.T(). We also can't copy the struct users embed suite.Suite in because they can depend on any other fields they care to set. Then other functions like Suite.Require and Suite.Assert are also unsafe for concurrent calls. I'm a bit stumped, can you raise a new issue for this?

Aniruddha Maru · Answer 11 · Sat Jul 31 2021 02:34:36 GMT+0800 (China Standard Time)

That should stop us from calling teardowns early, do you agree?

Nice, yeah I think that works.

Aniruddha Maru · Answer 12 · Sat Jul 31 2021 02:47:56 GMT+0800 (China Standard Time)

Nice, yeah I think that works.

Never mind, spoke too soon, it leads to a deadlock because parallel sub-tests wait for parent test to complete, and parent test won't complete if its waiting for child tests to be done. I'll dig further, however: if parallel tests can't be run, this becomes a moot point. I'd rather document that Suite sub-tests shouldn't call s.T().Parallel() and leave it at that.

I'm a bit stumped, can you raise a new issue for this?

Yep, same, there has been some discussion in #187 on this very topic which covers what we talked about here. I suspect this will require a breaking change in the API. I'll do some more digging on this as well.

Bracken · Answer 13 · Sat Jul 31 2021 04:48:36 GMT+0800 (China Standard Time)

I'd rather document that Suite sub-tests shouldn't call s.T().Parallel() and leave it at that.

That might be best, what a gnarly issue.

I suspect this will require a breaking change in the API.

I think you're right, perhaps we can come up with a workable design that supports parallel testing for v2.0? Though I must admit I never actually use suite.

Aniruddha Maru · Answer 14 · Sat Jul 31 2021 05:59:22 GMT+0800 (China Standard Time)

Fwiw, here's a couple of possible fixes for the tearDown ordering:

Group subtests under a single group within the test suite: master...maroux:group_tests (go 1.6+)
Use T.Cleanup: master...maroux:parallel_tests_cleanup (go 1.14+)

Although, as mentioned, I think we should just close this ticket and document that parallel testing is not supported for now.

Aniruddha Maru · Answer 15 · Sat Jul 31 2021 06:49:53 GMT+0800 (China Standard Time)

workable design that supports parallel testing for v2.0

I imagine the easiest way is to allow test sub-methods to have a signature like so:

(_ *FooSuite) test_foo(s *suite.SuiteHelper) {
     s.Equal(...)
}

and move T, Assertions and Require out of Suite into a separate struct (named SuiteHelper) above.

Aniruddha Maru · Answer 16 · Sat Jul 31 2021 13:00:55 GMT+0800 (China Standard Time)

Alright, master...maroux:suite_copy also shows a possible implementation for concurrency-safe suite.

Kirusha · Answer 17 · Sat Aug 17 2024 16:35:01 GMT+0800 (China Standard Time)

Any solution? It still doesn't work properly.

Anton Telyshev · Answer 18 · Sun Aug 25 2024 23:01:57 GMT+0800 (China Standard Time)

Covered by testifylint#suite-broken-parallel 👌

Bracken · Answer 19 · Wed Oct 02 2024 06:41:12 GMT+0800 (China Standard Time)

#1109 shows how suite can be made to support parallel tests. As this is a breaking change, there is no plan to merge it into this testify. We are maintaining the repo at v1 only. I do not see a good way of fixing this without the breaking change.

Those who wish to use suite and parallel tests should use a fork including the work in #1109. Those who wish to use suite from this repository should not use parallel tests. Suite does not support parallel tests.