Flake: TestRefreshingCertificate panics
m1kola opened this issue · comments
panic: Log in goroutine after TestRefreshingCertificate has completed
goroutine 8 [running]:
testing.(*common).logDepth(0xc0002c39e0, 0xc0000b2000, 0x17e, 0x3)
/usr/local/go/src/testing/testing.go:730 +0x581
testing.(*common).log(...)
/usr/local/go/src/testing/testing.go:712
testing.(*common).Fatalf(0xc0002c39e0, 0x150c996, 0x2e, 0xc0000ac140, 0x5, 0x5)
/usr/local/go/src/testing/testing.go:780 +0x82
github.com/golang/mock/gomock.(*Controller).Call.func1(0xc0003214d0, 0x140c2a0, 0xc0000522a0, 0x14ec810, 0x14, 0xc00000e460, 0x2, 0x2, 0x0, 0x0, ...)
/home/cloud-user/agent/_work/1/go/src/github.com/Azure/ARO-RP/vendor/github.com/golang/mock/gomock/controller.go:231 +0x485
github.com/golang/mock/gomock.(*Controller).Call(0xc0003214d0, 0x140c2a0, 0xc0000522a0, 0x14ec810, 0x14, 0xc00000e460, 0x2, 0x2, 0x0, 0x1, ...)
/home/cloud-user/agent/_work/1/go/src/github.com/Azure/ARO-RP/vendor/github.com/golang/mock/gomock/controller.go:247 +0xb4
github.com/Azure/ARO-RP/pkg/util/mocks/keyvault.(*MockManager).GetCertificateSecret(0xc0000522a0, 0x16fafa0, 0xc000120010, 0x14e74e7, 0xf, 0xc00004a420, 0xc000071f78, 0xc000390240, 0xc0002d1e88, 0xc0002d1f38, ...)
/home/cloud-user/agent/_work/1/go/src/github.com/Azure/ARO-RP/pkg/util/mocks/keyvault/keyvault.go:88 +0x131
github.com/Azure/ARO-RP/pkg/env.(*refreshingCertificate).fetchCertificateOnce(0xc0000a8150, 0x16fafa0, 0xc000120010, 0x0, 0x0)
/home/cloud-user/agent/_work/1/go/src/github.com/Azure/ARO-RP/pkg/env/certificateRefresher.go:69 +0x9a
github.com/Azure/ARO-RP/pkg/env.(*refreshingCertificate).fetchCertificate.func1(0xc0000ac0a0, 0xc0000a8150, 0x16fafa0, 0xc000120010)
/home/cloud-user/agent/_work/1/go/src/github.com/Azure/ARO-RP/pkg/env/certificateRefresher.go:93 +0x144
created by github.com/Azure/ARO-RP/pkg/env.(*refreshingCertificate).fetchCertificate
/home/cloud-user/agent/_work/1/go/src/github.com/Azure/ARO-RP/pkg/env/certificateRefresher.go:86 +0x79
FAIL github.com/Azure/ARO-RP/pkg/env 0.257s
For visibiltiy: had a chat with @petrkotas. This looks like a different sympthom of the issue which is being fixed in #1636.
@petrkotas I think there's still a flake here. On unit test run of #1645 it failed the first time which had your changes from #1636. I reran and it worked successfully. Tried to reproduce locally but I couldn't.
Look at build run 1 (number 2 passed successfully) here. You might have to click "Show Previous Attempts".
=== FAIL: pkg/env TestRefreshingCertificate/test_refresh_certificate,_pull_exactly_twice,_first_on_start,_second_on_refresh (0.06s)
certificateRefresher_test.go:272: returned private key does not match
certificateRefresher_test.go:276: returned certificate does not match
--- FAIL: TestRefreshingCertificate/test_refresh_certificate,_pull_exactly_twice,_first_on_start,_second_on_refresh (0.06s)
We are currently using time to do these tests. Maybe we could use clockwork or something that mocks time to get from what seems to be a race.
FWIW: I ran into this a few times recently in PR's, so attempted to recreate it locally. I ran the unit tests for over 2 hours non-stop with 130 different attempts without failure. This makes me more suspicious of the azure environment since I could not reproduce locally, but will continue to attempt to do so.
Nevermind, as I posted this I was able to get the error locally. Good news.