Data race between flagsaver and OperationManager

[Martin Hutchinson]

From a cursory glance it looks like this is test-only code. Manifested on 2/5 runs. Was first detected in Travis when I was modifying the config which caused more tests to be covered by race than normal (https://travis-ci.org/google/trillian/jobs/589040627).

To repro:
go test -count 5 -race -run TestNewClientDialOptionsFromFlagsWithTLSCertFileSet ./client/rpcflags

Result:

WARNING: DATA RACE
Write at 0x000001dc2640 by goroutine 123:
  flag.(*boolValue).Set()
      /usr/lib/google-golang/src/flag/flag.go:121 +0x7f
  flag.(*FlagSet).Set()
      /usr/lib/google-golang/src/flag/flag.go:426 +0x111
  github.com/google/trillian/testonly/flagsaver.(*Stash).Restore()
      /usr/lib/google-golang/src/flag/flag.go:439 +0x158
  github.com/google/trillian/testonly/flagsaver.(*Stash).MustRestore()
      /usr/local/google/home/mhutchinson/gitclients/trillian/testonly/flagsaver/flagsaver.go:69 +0x3f
  github.com/google/trillian/client/rpcflags.TestNewClientDialOptionsFromFlagsWithTLSCertFileSet()
      /usr/local/google/home/mhutchinson/gitclients/trillian/client/rpcflags/rpcflags_test.go:135 +0xa7d
  testing.tRunner()
      /usr/lib/google-golang/src/testing/testing.go:909 +0x199

Previous read at 0x000001dc2640 by goroutine 134:
  github.com/golang/glog.(*loggingT).output()
      /usr/local/google/home/mhutchinson/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:682 +0x17d
  github.com/golang/glog.(*loggingT).printDepth()
      /usr/local/google/home/mhutchinson/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:646 +0x22e
  github.com/google/trillian/log.(*logOperationExecutor).run()
      /usr/local/google/home/mhutchinson/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:637 +0x6b4
  github.com/google/trillian/log.(*OperationManager).getLogsAndExecutePass()
      /usr/local/google/home/mhutchinson/gitclients/trillian/log/operation_manager.go:306 +0x4b2
  github.com/google/trillian/log.(*OperationManager).OperationLoop()
      /usr/local/google/home/mhutchinson/gitclients/trillian/log/operation_manager.go:327 +0x4fe
  github.com/google/trillian/testonly/integration.NewLogEnvWithRegistryAndGRPCOptions.func1()
      /usr/local/google/home/mhutchinson/gitclients/trillian/testonly/integration/logenv.go:148 +0x80

Goroutine 123 (running) created at:
  testing.(*T).Run()
      /usr/lib/google-golang/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/lib/google-golang/src/testing/testing.go:1204 +0xa6
  testing.tRunner()
      /usr/lib/google-golang/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/lib/google-golang/src/testing/testing.go:1202 +0x521
  testing.(*M).Run()
      /usr/lib/google-golang/src/testing/testing.go:1119 +0x309
  main.main()
      _testmain.go:48 +0x223

Goroutine 134 (running) created at:
  github.com/google/trillian/testonly/integration.NewLogEnvWithRegistryAndGRPCOptions()
      /usr/local/google/home/mhutchinson/gitclients/trillian/testonly/integration/logenv.go:146 +0x686
  github.com/google/trillian/testonly/integration.NewLogEnvWithGRPCOptions()
      /usr/local/google/home/mhutchinson/gitclients/trillian/testonly/integration/logenv.go:99 +0x3f8
  github.com/google/trillian/client/rpcflags.TestNewClientDialOptionsFromFlagsWithTLSCertFileSet()
      /usr/local/google/home/mhutchinson/gitclients/trillian/client/rpcflags/rpcflags_test.go:105 +0x550
  testing.tRunner()
      /usr/lib/google-golang/src/testing/testing.go:909 +0x199
==================
--- FAIL: TestNewClientDialOptionsFromFlagsWithTLSCertFileSet (0.58s)
    testing.go:853: race detected during execution of test