metamorphic: assignment to entry in nil map
RaduBerinde opened this issue · comments
I found this failure case with TestMetaTwoInstance:
Init(2 /* dbs */, 45 /* batches */, 67 /* iters */, 48 /* snapshots */)
db1.RangeKeySet("a", "c", "@10", "")
db1.Flush()
iter64 = db1.NewIter("", "", 2 /* key types */, 0, 0, false /* use L6 filters */, "" /* masking suffix */)
iter64.SeekGE("b", "")
db1.Close()
OPTIONS: https://gist.github.com/RaduBerinde/b2bb75ea16b178d6881edfe66d547935
=== RUN TestMetaTwoInstance
// INFO: [JOB 1] MANIFEST created 000001
// INFO: [JOB 1] WAL created 000002
// db1.Open() <nil>
// INFO: [JOB 1] MANIFEST created 000001
// INFO: [JOB 2] all initial table stats loaded
// INFO: [JOB 1] WAL created 000002
// db2.Open() <nil>
Init(2 /* dbs */, 45 /* batches */, 67 /* iters */, 48 /* snapshots */) #0
// INFO: [JOB 2] all initial table stats loaded
db1.RangeKeySet("a", "c", "@10", "") // <nil> #1
// INFO: [JOB 3] WAL created 000004
// INFO: [JOB 4] flushing 1 memtable (43B) to L0
// INFO: [JOB 4] flushing: sstable created 000005
// INFO: [JOB 4] flushed 1 memtable (43B) to L0 [000005] (928B), in 0.0s (0.0s total), output rate 3.0MB/s
db1.Flush() // <nil> #2
iter64 = db1.NewIter("", "", 2 /* key types */, 0, 0, false /* use L6 filters */, "" /* masking suffix */) // <nil> #3
// INFO: [JOB 4] WAL deleted 000002
db1.Close() // leaked iterators: current
0.0:
000005:[a#10,RANGEKEYSET-c#inf,RANGEKEYSET] #5
panic: assignment to entry in nil map
goroutine 58 [running]:
github.com/cockroachdb/pebble.(*tableCacheShard).addNode(0x1400027a000, 0x1400029a570, 0x1400020e488)
/Users/radu/go/src/github.com/cockroachdb/pebble/table_cache.go:897 +0x8c
github.com/cockroachdb/pebble.(*tableCacheShard).findNodeInternal(0x1400027a000, 0x14000192a80, 0x1400020e488)
/Users/radu/go/src/github.com/cockroachdb/pebble/table_cache.go:828 +0x3b4
github.com/cockroachdb/pebble.(*tableCacheShard).findNode(...)
/Users/radu/go/src/github.com/cockroachdb/pebble/table_cache.go:778
github.com/cockroachdb/pebble.(*tableCacheShard).newRangeKeyIter(0x1400027a000, 0x14000192a80, {{0x0?, 0x1?, 0x140002b35d8?}}, 0x1400020e488)
/Users/radu/go/src/github.com/cockroachdb/pebble/table_cache.go:596 +0x58
github.com/cockroachdb/pebble.(*tableCacheContainer).newRangeKeyIter(0x140002b3618?, 0x100cf50b0?, {{0x0?, 0x0?, 0x1?}})
/Users/radu/go/src/github.com/cockroachdb/pebble/table_cache.go:150 +0x4c
github.com/cockroachdb/pebble/internal/keyspan.(*LevelIter).loadFile(0x140002cc778, 0x140002aa050?, 0x14000240250?)
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/level_iter.go:182 +0x150
github.com/cockroachdb/pebble/internal/keyspan.(*LevelIter).SeekLT(0x140002cc778, {0x14000240250, 0x1, 0x8})
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/level_iter.go:284 +0x2cc
github.com/cockroachdb/pebble/internal/keyspan.(*MergingIter).SeekGE(0x140002cc2e8, {0x14000240250, 0x1, 0x8})
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/merging_iter.go:369 +0xc0
github.com/cockroachdb/pebble/internal/keyspan.(*BoundedIter).SeekGE(0x140002cc668, {0x14000240250?, 0x0?, 0x0?})
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/bounded.go:104 +0x34
github.com/cockroachdb/pebble/internal/keyspan.(*DefragmentingIter).SeekGE(0x140002cc6d8, {0x14000240250, 0x1, 0x8})
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/defragment.go:212 +0x3c
github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).keyspanSeekGE(0x140002cc028, {0x14000240250?, 0x0?, 0x0?}, {0x0?, 0x0?, 0x0?})
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/interleaving_iter.go:824 +0x34
github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).InitSeekGE(0x140002cc028, {0x0, 0x0, 0x0}, {0x14000240250, 0x1, 0x8}, 0x0?, {{0x0?, 0x100e8458c?, ...}, ...})
/Users/radu/go/src/github.com/cockroachdb/pebble/internal/keyspan/interleaving_iter.go:241 +0x110
github.com/cockroachdb/pebble.(*lazyCombinedIter).initCombinedIteration(0x14000498458, 0x1, 0x0, {{0x0?, 0x0?, 0x0?}, 0x0?}, {0x14000240250?, 0x140002a5b98?, 0x1009598cc?})
/Users/radu/go/src/github.com/cockroachdb/pebble/range_keys.go:579 +0x444
github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekGE(0x14000498458, {0x14000240250, 0x1, 0x8}, 0x0?)
/Users/radu/go/src/github.com/cockroachdb/pebble/range_keys.go:592 +0x80
github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit(0x14000498000, {0x14000240250, 0x1, 0x8}, {0x0, 0x0, 0x0})
/Users/radu/go/src/github.com/cockroachdb/pebble/iterator.go:1319 +0x56c
github.com/cockroachdb/pebble.(*Iterator).SeekGE(...)
/Users/radu/go/src/github.com/cockroachdb/pebble/iterator.go:1203
github.com/cockroachdb/pebble/metamorphic.(*retryableIter).SeekGE.func1()
/Users/radu/go/src/github.com/cockroachdb/pebble/metamorphic/retryable.go:139 +0x44
github.com/cockroachdb/pebble/metamorphic.(*retryableIter).withRetry(0x14000151900, 0x140002b3d38)
/Users/radu/go/src/github.com/cockroachdb/pebble/metamorphic/retryable.go:39 +0x34
github.com/cockroachdb/pebble/metamorphic.(*retryableIter).SeekGE(0x140002a5e88?, {0x14000240250?, 0x140002a5f38?, 0x2?})
/Users/radu/go/src/github.com/cockroachdb/pebble/metamorphic/retryable.go:139 +0x58
github.com/cockroachdb/pebble/metamorphic.(*iterSeekGEOp).run(0x1400024c700, 0x140002a5f00?, {0x140001511a0?, 0x0?})
/Users/radu/go/src/github.com/cockroachdb/pebble/metamorphic/ops.go:1197 +0xc0
github.com/cockroachdb/pebble/metamorphic.Execute.func1()
/Users/radu/go/src/github.com/cockroachdb/pebble/metamorphic/meta.go:556 +0x134
golang.org/x/sync/errgroup.(*Group).Go.func1()
/Users/radu/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 20
/Users/radu/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:72 +0x98
FAIL github.com/cockroachdb/pebble/internal/metamorphic 0.411s
FAIL
Ah, this is more of an issue with the metamorphic test, combined with a not-so-nice sequence of operations which leaks iterators. We do the Seek and the close in parallel from different threads (because they use different receiver objects).