github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed
cockroach-teamcity opened this issue · comments
github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ e5c9f635e505:
// INFO: [JOB 661] flushing: sstable created 000571
// INFO: [JOB 661] MANIFEST created 000572
// INFO: [JOB 661] flushed 1 memtable (173B) to L0 [000571] (796B), in 0.0s (0.0s total), output rate 1.1MB/s
// INFO: [JOB 661] MANIFEST deleted 000547
// INFO: [JOB 663] compacting(default) L0 [000560 000571] (1.5KB) Score=100.00 + L5 [] (0B) Score=0.00; OverlappingRatio: Single 0.00, Multi 0.00
// INFO: [JOB 663] compacting: sstable created 000573
// INFO: [JOB 663] compacted(default) L0 [000560 000571] (1.5KB) Score=100.00 + L5 [] (0B) Score=0.00 -> L5 [000573] (893B), in 0.0s (0.0s total), output rate 90KB/s
// INFO: [JOB 665] compacting(default) L5 [000573 000568] (1.6KB) Score=24.63 + L6 [000567] (1.1KB) Score=0.26; OverlappingRatio: Single 0.68, Multi 0.00
// INFO: [JOB 663] sstable deleted 000571
// INFO: [JOB 665] compacting: sstable created 000574
// INFO: [JOB 665] compacted(default) L5 [000573 000568] (1.6KB) Score=24.63 + L6 [000567] (1.1KB) Score=0.26 -> L6 [000574] (1.1KB), in 0.0s (0.0s total), output rate 453KB/s
// INFO: [JOB 666] compacting(default) L0 [000570 000569] (1.5KB) Score=100.00 + L5 [] (0B) Score=0.00; OverlappingRatio: Single 0.00, Multi 0.00
// INFO: [JOB 665] sstable deleted 000567
// INFO: [JOB 660] ingested L0:000570 (776B), L0:000569 (776B), L5:000568 (776B)
db1.IngestExternalFiles(external0, "utlyzfoxv" /* start */, "utlyzfoxvduv" /* end */, "" /* syntheticSuffix */, "" /* syntheticPrefix */, external0, "pfnjutlyzfoxv" /* start */, "pfnjutlyzfoxvkmkydajc" /* end */, "" /* syntheticSuffix */, "pfnj" /* syntheticPrefix */, external0, "cautlyzfoxv" /* start */, "cautlyzfoxvbx" /* end */, "@148" /* syntheticSuffix */, "ca" /* syntheticPrefix */) // <nil> #2443
// INFO: [JOB 666] compacting: sstable created 000575
db1.Set("bdqj@38", "glfqfvpse") // <nil> #2444
// INFO: [JOB 665] sstable deleted 000573
// INFO: [JOB 666] compacted(default) L0 [000570 000569] (1.5KB) Score=100.00 + L5 [] (0B) Score=0.00 -> L5 [000575] (777B), in 0.0s (0.0s total), output rate 413KB/s
db1.Get("bdqj@29") // [""] pebble: not found #2445
db1.IngestExternalFiles(external0, "utlyzfoxvfyzrtgrw" /* start */, "utlyzfoxvlaj" /* end */, "@149" /* syntheticSuffix */, "" /* syntheticPrefix */) // no-op #2450
// INFO: [JOB 669] compacting(default) L5 [000575] (777B) Score=10.43 + L6 [000574] (1.1KB) Score=0.61; OverlappingRatio: Single 1.49, Multi 0.00
// INFO: [JOB 669] compacting: sstable created 000578
// INFO: [JOB 668] ingesting: sstable created 000577
// INFO: [JOB 669] compacted(default) L5 [000575] (777B) Score=10.43 + L6 [000574] (1.1KB) Score=0.61 -> L6 [000578] (1.1KB), in 0.0s (0.0s total), output rate 165KB/s
// INFO: [JOB 669] sstable deleted 000574
// INFO: [JOB 671] compacting(default) L5 [000576] (776B) Score=7.00 + L6 [000578] (1.1KB) Score=0.43; OverlappingRatio: Single 1.49, Multi 0.00
// INFO: [JOB 669] sstable deleted 000575
// INFO: [JOB 668] ingested L0:000577 (776B), L5:000576 (776B)
db1.IngestExternalFiles(external0, "utlyzfoxv" /* start */, "utlyzfoxvex" /* end */, "" /* syntheticSuffix */, "" /* syntheticPrefix */, external0, "kimutlyzfoxv" /* start */, "kimutlyzfoxvet" /* end */, "" /* syntheticSuffix */, "kim" /* syntheticPrefix */) // <nil> #2451
// INFO: [JOB 671] compacting: sstable created 000579
db1.Merge("szle@78", "namqusdtlhgwp") // <nil> #2452
db1.Set("ozfb@139", "xygmodpnsyvct") // <nil> #2453
db1.Get("yconcminh@107") // [""] pebble: not found #2455
db1.DeleteRange("kllhmbsxunem@21", "tiiirmn@74") // <nil> #2456
db1.Merge("hmmwvdxe@83", "") // <nil> #2457
iter19.Close() // <nil> #2458
// INFO: [JOB 672] sstable deleted 000551
// INFO: [JOB 671] MANIFEST created 000580
// INFO: [JOB 671] compacted(default) L5 [000576] (776B) Score=7.00 + L6 [000578] (1.1KB) Score=0.43 -> L6 [000579] (1.1KB), in 0.0s (0.0s total), output rate 505KB/s
// INFO: [JOB 673] ingested L5:000581 (776B)
db1.IngestExternalFiles(external0, "utlyzfoxv" /* start */, "utlyzfoxvd" /* end */, "@151" /* syntheticSuffix */, "" /* syntheticPrefix */, external0, "txvkoutlyzfoxvfyzrtgrw" /* start */, "txvkoutlyzfoxvmugd" /* end */, "@150" /* syntheticSuffix */, "txvko" /* syntheticPrefix */) // <nil> #2463
db1.SingleDelete("gltzry@55", true /* maybeReplaceDelete */) // <nil> #2464
// INFO: [JOB 671] sstable deleted 000578
// INFO: [JOB 671] MANIFEST deleted 000562
iter18.Close() // <nil> #2465
// INFO: [JOB 677] compacting(default) L5 [000581] (776B) Score=7.00 + L6 [000579] (1.1KB) Score=0.43; OverlappingRatio: Single 1.49, Multi 0.00
To reduce: go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --run-dir /artifacts/meta/240322-043500.583754581672/random-009 --try-to-reduce -v
--- FAIL: TestMeta/execution/random-009 (4.82s)
Help
To reproduce, try:
go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1711082100583875351 -ops "uniform:5000-10000"
A data race. @RaduBerinde maybe related to the backing file refactors?
WARNING: DATA RACE
Write at 0x00c0001b7c00 by goroutine 26:
slices.DeleteFunc[go.shape.[]github.com/cockroachdb/pebble/internal/base.DiskFileNum,go.shape.uint64]()
GOROOT/src/slices/slices.go:242 +0x159
github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*remoteLockedState).removeExternalObject()
github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/remote.go:70 +0x195
github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).removeMetadata()
github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:515 +0x27a
github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).Remove()
github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:293 +0x278
github.com/cockroachdb/pebble.(*cleanupManager).deleteObsoleteObject()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:271 +0x1f4
github.com/cockroachdb/pebble.(*cleanupManager).mainLoop()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:167 +0x37e
github.com/cockroachdb/pebble.openCleanupManager.func1.1()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:96 +0x2e
runtime/pprof.Do()
GOROOT/src/runtime/pprof/runtime.go:51 +0x117
github.com/cockroachdb/pebble.openCleanupManager.func1()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:95 +0x91
Previous read at 0x00c0001b7c00 by goroutine 43:
github.com/cockroachdb/pebble.(*DB).findExistingBackingsForExternalObjects()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:733 +0x306
github.com/cockroachdb/pebble.(*DB).ingestAttachRemote()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:639 +0x387
github.com/cockroachdb/pebble.(*DB).ingest()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:1492 +0x947
github.com/cockroachdb/pebble.(*DB).IngestExternalFiles()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:1271 +0x185
github.com/cockroachdb/pebble/metamorphic.(*ingestExternalFilesOp).run()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/ops.go:1047 +0x8f7
github.com/cockroachdb/pebble/metamorphic.Execute.func1()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/meta.go:612 +0x28c
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91
Goroutine 26 (running) created at:
github.com/cockroachdb/pebble.openCleanupManager()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/cleaner.go:94 +0x4fc
github.com/cockroachdb/pebble.Open()
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:387 +0x3757
github.com/cockroachdb/pebble/metamorphic.(*Test).init.func12()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/test.go:183 +0x8f
github.com/cockroachdb/pebble/metamorphic.withRetries()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/retryable.go:29 +0x2e
github.com/cockroachdb/pebble/metamorphic.(*Test).withRetries()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/test.go:240 +0x1c64
github.com/cockroachdb/pebble/metamorphic.(*Test).init()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/test.go:182 +0x1b6b
github.com/cockroachdb/pebble/metamorphic.RunOnce()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/meta.go:547 +0x2134
github.com/cockroachdb/pebble/internal/metamorphic.runTestMeta()
github.com/cockroachdb/pebble/internal/metamorphic/external/com_github_cockroachdb_pebble/internal/metamorphic/meta_test.go:87 +0x657
github.com/cockroachdb/pebble/internal/metamorphic.TestMeta()
github.com/cockroachdb/pebble/internal/metamorphic/external/com_github_cockroachdb_pebble/internal/metamorphic/meta_test.go:54 +0x28
testing.tRunner()
GOROOT/src/testing/testing.go:1689 +0x21e
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:1742 +0x44
Goroutine 43 (running) created at:
golang.org/x/sync/errgroup.(*Group).Go()
golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x124
github.com/cockroachdb/pebble/metamorphic.Execute()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/meta.go:589 +0xa8
github.com/cockroachdb/pebble/metamorphic.RunOnce()
github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/meta.go:549 +0x21b3
github.com/cockroachdb/pebble/internal/metamorphic.runTestMeta()
github.com/cockroachdb/pebble/internal/metamorphic/external/com_github_cockroachdb_pebble/internal/metamorphic/meta_test.go:87 +0x657
github.com/cockroachdb/pebble/internal/metamorphic.TestMeta()
github.com/cockroachdb/pebble/internal/metamorphic/external/com_github_cockroachdb_pebble/internal/metamorphic/meta_test.go:54 +0x28
testing.tRunner()
GOROOT/src/testing/testing.go:1689 +0x21e
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:1742 +0x44
==================
Yeah it's related to the backing reuse change.