scylladb / gocql

Package gocql implements a fast and robust ScyllaDB client for the Go programming language.

Home Page:https://docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-go-driver.html

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multiple session during go tests, resulting in failed tests

alexvnovelo opened this issue · comments

Please answer these questions before submitting your issue. Thanks!

Background:
I'm running golang integration tests that connect to one of our staging environments scylladb clusters. When i run go test ./... it actually runs 2 packages in parallel (sub directories from root directory where i called go test ./...). In each of these packages, there is one main _test.go file that runs through a suite of tests, creating their own session but under the same configurations. The close separately once their tests have finished.

When i run each of these packages seperately, the work perfectly. All tests pass, no timeouts, etc. However, when I run them together, I get the failing tests in both (1 failed test for each package), and occasionally the stack trace being printed out with the goroutines as shown in the "### What did you see instead?" section.

So my question is, Is it bad to have multiple session when running go tests? It seems to only have these issues when both of these sessions are open. Any help would be appreciated!

What version of Scylla or Cassandra are you using?

Scylla Enterprise 2021.1.8

What version of Gocql are you using?

github.com/gocql/gocql v0.0.0-20191102131523

What version of Go are you using?

go version go1.17.5 darwin/amd64

What did you do?

Ran integration tests

What did you expect to see?

Passing tests

What did you see instead?

goroutine 85 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000382f00, 0x30d40)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:814 +0x139
created by github.com/gocql/gocql.newWriteCoalescer
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:719 +0x125

goroutine 83 [IO wait]:
internal/poll.runtime_pollWait(0x7f96effc9310, 0x72, 0xffffffffffffffff)
/opt/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0003a0398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/opt/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/opt/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0003a0380, 0xc000346000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0003a0380, 0xc000346000, 0x1000, 0x1000, 0xc0003a7080, 0x3, 0xc0003ade88)
/opt/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0002c8300, 0xc000346000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:183 +0x91
bufio.(*Reader).Read(0xc0002819e0, 0xc00033e228, 0x1, 0x9, 0x40, 0x38, 0xce7ea0)
/opt/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0xe23ec0, 0xc0002819e0, 0xc00033e228, 0x1, 0x9, 0x1, 0xc0003add70, 0x410058, 0x38)
/opt/go/src/io/io.go:328 +0x87
io.ReadFull(...)
/opt/go/src/io/io.go:347
github.com/gocql/gocql.readHeader(0xe23ec0, 0xc0002819e0, 0xc00033e228, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/frame.go:449 +0x96
github.com/gocql/gocql.(*Conn).recv(0xc00033e1e0, 0x0, 0x0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:593 +0xfd
github.com/gocql/gocql.(*Conn).serve(0xc00033e1e0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:510 +0x31
created by github.com/gocql/gocql.(*Session).dialWithoutObserver
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:283 +0x6ca

goroutine 86 [IO wait]:
internal/poll.runtime_pollWait(0x7f96effc93f8, 0x72, 0xffffffffffffffff)
/opt/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00012a818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/opt/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/opt/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00012a800, 0xc000348000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc00012a800, 0xc000348000, 0x1000, 0x1000, 0xc0003a7200, 0x3, 0xc000082e88)
/opt/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0002c8310, 0xc000348000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:183 +0x91
bufio.(*Reader).Read(0xc000281c20, 0xc00033e318, 0x1, 0x9, 0x40, 0x38, 0xce7ea0)
/opt/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0xe23ec0, 0xc000281c20, 0xc00033e318, 0x1, 0x9, 0x1, 0xc000082d70, 0x410058, 0x38)
/opt/go/src/io/io.go:328 +0x87
io.ReadFull(...)
/opt/go/src/io/io.go:347
github.com/gocql/gocql.readHeader(0xe23ec0, 0xc000281c20, 0xc00033e318, 0x9, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/frame.go:449 +0x96
github.com/gocql/gocql.(*Conn).recv(0xc00033e2d0, 0x0, 0x0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:593 +0xfd
github.com/gocql/gocql.(*Conn).serve(0xc00033e2d0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:510 +0x31
created by github.com/gocql/gocql.(*Session).dialWithoutObserver
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:283 +0x6ca

goroutine 84 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc00033e1e0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:550 +0x110
created by github.com/gocql/gocql.(*Session).dialWithoutObserver
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:284 +0x6ef

goroutine 87 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc00033e2d0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:550 +0x110
created by github.com/gocql/gocql.(*Session).dialWithoutObserver
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:284 +0x6ef

goroutine 99 [select]:
github.com/gocql/gocql.(*Conn).exec(0xc0004280f0, 0xe35860, 0xc000125140, 0xe24700, 0xc0002ca300, 0x0, 0x0, 0x0, 0x0, 0x0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:902 +0x3d3
github.com/gocql/gocql.(*Conn).executeQuery(0xc0004280f0, 0xe35860, 0xc000125140, 0xc0001f6240, 0x2a89755125acdb)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:1121 +0x569
github.com/gocql/gocql.(*Query).execute(0xc0001f6240, 0xe35860, 0xc000125140, 0xc0004280f0, 0x6be717)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/session.go:949 +0x49
github.com/gocql/gocql.(*queryExecutor).attemptQuery(0xc0003802a0, 0xe35860, 0xc000125140, 0xe3e5d0, 0xc0001f6240, 0xc0004280f0, 0xc0003ca000)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/query_executor.go:29 +0x84
github.com/gocql/gocql.(*queryExecutor).do(0xc0003802a0, 0xe35860, 0xc000125140, 0xe3e5d0, 0xc0001f6240, 0x40f8fb)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/query_executor.go:112 +0x1c7
github.com/gocql/gocql.(*queryExecutor).executeQuery(0xc0003802a0, 0xe3e5d0, 0xc0001f6240, 0x0, 0x0, 0x0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/query_executor.go:60 +0xfb
github.com/gocql/gocql.(*Session).executeQuery(0xc00039ea80, 0xc0001f6240, 0xd36216)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/session.go:426 +0xb2
github.com/gocql/gocql.(*Query).Iter(0xc0001f6240, 0xc0001f6240)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/session.go:1130 +0x45
github.com/gocql/gocql.(*Query).Exec(0xc0001f6240, 0xc0001f6240, 0x1e)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/session.go:1113 +0x2b
medium.com/pkg/cql.(*Client).Exec(0xc000010308, 0xe35860, 0xc000125110, 0xe25820, 0xc00000e978, 0x0, 0x0)
/workspace/go/pkg/cql/client.go:84 +0x1e5
medium.com/cmd/store/scylla.(*Store).DropTables(0xc00022c0e0, 0xe357f0, 0xc00003c0a8, 0xc0002cd790, 0x9, 0x0, 0x0)
/workspace/go/cmd/store/scylla/tables.go:324 +0x17e
medium.com/cmd/store/scylla.testFeatures(0xc000103c80, 0xe357f0, 0xc00003c0a8, 0xc00022c0e0)
/workspace/go/cmd/store/scylla/entity_test.go:118 +0x1665
medium.com/cmd/store/scylla.TestStore.func1.1(0xc000103c80)
/workspace/go/cmd/store/scylla/store_test.go:18 +0x4a
testing.tRunner(0xc000103c80, 0xc000325920)
/opt/go/src/testing/testing.go:1194 +0xef
created by testing.(*T).Run
/opt/go/src/testing/testing.go:1239 +0x2b3

goroutine 11 [select, 2 minutes]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc00039ea80, 0xdf8475800)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/session.go:286 +0x1b1
created by github.com/gocql/gocql.(*Session).init
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/session.go:252 +0x6df

goroutine 47 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00034a2a0, 0x30d40)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:814 +0x139
created by github.com/gocql/gocql.newWriteCoalescer
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:719 +0x125

goroutine 49 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc0004280f0)
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:550 +0x110
created by github.com/gocql/gocql.(*Session).dialWithoutObserver
/builder/home/go/pkg/mod/github.com/gocql/gocql@v0.0.0-20191102131523-9faa4c08d962/conn.go:284 +0x6ef

If you are having connectivity related issues please share the following additional information

Describe your Cassandra cluster

please provide the following information

  • output of nodetool status
  • output of SELECT peer, rpc_address FROM system.peers
  • rebuild your application with the gocql_debug tag and post the output

Do you see a panic or is there something else that causes the stack trace to be printed?

github.com/gocql/gocql v0.0.0-20191102131523

This version seems to be old and seems to be the upstream gocql/gocql, not scylladb/gocql. Please try using the latest version of scylladb/gocql, see https://github.com/scylladb/gocql#installation

Do you see a panic or is there something else that causes the stack trace to be printed?

github.com/gocql/gocql v0.0.0-20191102131523

This version seems to be old and seems to be the upstream gocql/gocql, not scylladb/gocql. Please try using the latest version of scylladb/gocql, see https://github.com/scylladb/gocql#installation

It ends up panicking due to a timeout error with the tests exceeding 10 minutes. The 2 packages, when tested separately, take around 3 minutes. This leads me to believe there is some some sort of resource competition/data race going on .

Is is bad that we're using the upstream gocql/gocql instead of scylladb/gocql? Have their been issues reported when using gocql/gocql and Scylla databases? Thanks!