[BUG]: Panic from a lot of DELs and SETs with very large keys
nathanhack opened this issue · comments
What version of Badger are you using?
v4.2.0
But it also happened with v3.2103.2.
What version of Go are you using?
1.21.1
Have you tried reproducing the issue with the latest release?
None
What is the hardware spec (RAM, CPU, OS)?
128GiB, AMD Ryzen™ 9 5950X × 32, Fedora Linux 38 (Workstation Edition)
What steps will reproduce the bug?
running a bunch (several 100k) of DELs followed by SETs with large very large keys.
Expected behavior and actual result.
Expected: no panic.
Actual: a panic
panic: cannot grow buffer beyond 2 gigabytes
goroutine 1424649 [running]:
github.com/google/flatbuffers/go.(*Builder).growByteBuffer(...)
/homeDIR/go/pkg/mod/github.com/google/flatbuffers@v23.5.26+incompatible/go/builder.go:216
github.com/google/flatbuffers/go.(*Builder).Prep(0xc152a58c80, 0x4, 0x5b48)
/homeDIR/go/pkg/mod/github.com/google/flatbuffers@v23.5.26+incompatible/go/builder.go:270 +0x2b8
github.com/google/flatbuffers/go.(*Builder).CreateByteVector(0xc152a58c80, {0xc13207a000, 0x5b48, 0xc152a58b60?})
/homeDIR/go/pkg/mod/github.com/google/flatbuffers@v23.5.26+incompatible/go/builder.go:390 +0x45
github.com/dgraph-io/badger/v4/table.(*Builder).writeBlockOffset(0xc07bb0e000?, 0xc152a58c80, 0xc0202ec410, 0x47765e50?)
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:591 +0x3d
github.com/dgraph-io/badger/v4/table.(*Builder).writeBlockOffsets(0xc001a15b90, 0x43f38e?)
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:579 +0xc5
github.com/dgraph-io/badger/v4/table.(*Builder).buildIndex(0xc001a15b90, {0xc001574000, 0xfbe5, 0x1156a})
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:540 +0x106
github.com/dgraph-io/badger/v4/table.(*Builder).Done(0xc001a15b90)
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:450 +0x145
github.com/dgraph-io/badger/v4/table.CreateTable({0xc0791fa030, 0x16}, 0xc001a15b90)
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/table.go:258 +0x3d
github.com/dgraph-io/badger/v4.(*levelsController).subcompact.func4(0xc001a15b90, 0xc001a14090?)
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go:846 +0x109
created by github.com/dgraph-io/badger/v4.(*levelsController).subcompact in goroutine 1418647
/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go:836 +0x68f
Additional information
No response