After a get test I got fatal error: runtime: out of memory
Badb0yBadb0y opened this issue · comments
Hello,
I ran the test and after I got this:
`warp: Benchmark data written to "warp-remote-2021-12-01[023543]-yh78.csv.zst"
fatal error: runtime: out of memory
runtime stack:
runtime.throw({0xa2f5b9, 0x4c000000})
runtime/panic.go:1198 +0x71
runtime.sysMap(0xc37c400000, 0x429820, 0xc00010de90)
runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0xec86a0, 0x25ee5)
runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0xec86a0, 0x25ee5, 0x0, 0x0)
runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
runtime/mheap.go:913 +0x69
runtime.systemstack()
runtime/asm_amd64.s:383 +0x49
goroutine 55426 [running]:
runtime.systemstack_switch()
runtime/asm_amd64.s:350 fp=0xc013e82450 sp=0xc013e82448 pc=0x465d20
runtime.(*mheap).alloc(0x4bdca000, 0x25ee5, 0xe8, 0x1)
runtime/mheap.go:907 +0x73 fp=0xc013e824a0 sp=0xc013e82450 pc=0x425b53
runtime.(*mcache).allocLarge(0xc21f87e3d8, 0x4bdc9240, 0x8, 0x0)
runtime/mcache.go:227 +0x89 fp=0xc013e82500 sp=0xc013e824a0 pc=0x4167c9
runtime.mallocgc(0x4bdc9240, 0xa08d60, 0x1)
runtime/malloc.go:1082 +0x5c5 fp=0xc013e82580 sp=0xc013e82500 pc=0x40cd45
runtime.makeslice(0xc013e825d0, 0x6, 0xc06dda90c0)
runtime/slice.go:98 +0x52 fp=0xc013e825a8 sp=0xc013e82580 pc=0x44c572
github.com/minio/warp/pkg/bench.Operations.FilterByEndpoint({0xc09f200000, 0x7960ea, 0xc06dda90c0}, {0xc1d3c43e90, 0x2f})
github.com/minio/warp/pkg/bench/ops.go:409 +0x5a fp=0xc013e826d8 sp=0xc013e825a8 pc=0x87675a
github.com/minio/warp/pkg/aggregate.RequestAnalysisHostsSingleSized.func1({0xc1d3c43e90, 0x2f})
github.com/minio/warp/pkg/aggregate/requests.go:219 +0xa6 fp=0xc013e827c0 sp=0xc013e826d8 pc=0x88ff86
github.com/minio/warp/pkg/aggregate.RequestAnalysisHostsSingleSized·dwrap·8()
github.com/minio/warp/pkg/aggregate/requests.go:228 +0x31 fp=0xc013e827e0 sp=0xc013e827c0 pc=0x88feb1
runtime.goexit()
runtime/asm_amd64.s:1581 +0x1 fp=0xc013e827e8 sp=0xc013e827e0 pc=0x467e01
created by github.com/minio/warp/pkg/aggregate.RequestAnalysisHostsSingleSized
github.com/minio/warp/pkg/aggregate/requests.go:217 +0x1e8
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x0)
runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xa0b800)
sync/waitgroup.go:130 +0x71
github.com/minio/warp/pkg/aggregate.Aggregate({0xc2d3d22000, 0x7987f8, 0x91c600}, {0x0, 0xc09eb889b0, 0x0})
github.com/minio/warp/pkg/aggregate/aggregate.go:275 +0x9b8
github.com/minio/warp/cli.printAnalysis(0xc0000bbba0, {0xc2d3d22000, 0x7987f8, 0x91c600})
github.com/minio/warp/cli/analyze.go:265 +0x436
github.com/minio/warp/cli.runServerBenchmark(0xc0000bbba0)
github.com/minio/warp/cli/benchserver.go:208 +0xdec
github.com/minio/warp/cli.runBench(0xc0000bbba0, {0xafafc0, 0xc0000ce6c0})
github.com/minio/warp/cli/benchmark.go:103 +0x13a
github.com/minio/warp/cli.mainGet(0xc0000bbba0)
github.com/minio/warp/cli/get.go:82 +0x2ed
github.com/minio/cli.HandleAction({0x970bc0, 0xa59738}, 0x3)
github.com/minio/cli@v1.22.0/app.go:498 +0x5a
github.com/minio/cli.Command.Run({{0xa254a6, 0x3}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0xa2e5b2, 0x15}, {0x0, ...}, ...}, ...)
github.com/minio/cli@v1.22.0/command.go:225 +0x9f7
github.com/minio/cli.(*App).Run(0xc0000bb520, {0xc00001e090, 0x9, 0x9})
github.com/minio/cli@v1.22.0/app.go:261 +0x7ac
github.com/minio/warp/cli.Main({0xc00001e090, 0x9, 0x9})
github.com/minio/warp/cli/cli.go:85 +0x247
main.main()
github.com/minio/warp/main.go:27 +0x2e
goroutine 6 [chan receive, 5 minutes]:
github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc0000813e0, 0x0)
github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:36 +0x49
created by github.com/rjeczalik/notify.newNonrecursiveTree
github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:29 +0x16f
goroutine 7 [chan receive, 5 minutes]:
github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc0000813e0, 0xc000081380)
github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:81 +0x5a
created by github.com/rjeczalik/notify.newNonrecursiveTree
github.com/rjeczalik/notify@v0.9.2/tree_nonrecursive.go:30 +0x1c8
goroutine 8 [select, 5 minutes]:
github.com/minio/md5-simd.(*md5Server).process(0xc0001fea00, 0xc000081bc0)
github.com/minio/md5-simd@v1.1.2/md5-server_amd64.go:200 +0x2e7
created by github.com/minio/md5-simd.NewServer
github.com/minio/md5-simd@v1.1.2/md5-server_amd64.go:88 +0x330
goroutine 16 [select, 5 minutes]:
github.com/minio/md5-simd.(*md5Server).process(0xc0001fef00, 0xc000081d40)
github.com/minio/md5-simd@v1.1.2/md5-server_amd64.go:200 +0x2e7
created by github.com/minio/md5-simd.NewServer
github.com/minio/md5-simd@v1.1.2/md5-server_amd64.go:88 +0x330
goroutine 35 [select, 5 minutes]:
github.com/minio/md5-simd.(*md5Server).process(0xc00012a000, 0xc000114180)
github.com/minio/md5-simd@v1.1.2/md5-server_amd64.go:200 +0x2e7
created by github.com/minio/md5-simd.NewServer
github.com/minio/md5-simd@v1.1.2/md5-server_amd64.go:88 +0x330
goroutine 55427 [running]:
goroutine running on other thread; stack unavailable
created by github.com/minio/warp/pkg/aggregate.RequestAnalysisHostsSingleSized
github.com/minio/warp/pkg/aggregate/requests.go:217 +0x1e8
goroutine 55425 [running]:
goroutine running on other thread; stack unavailable
created by github.com/minio/warp/pkg/aggregate.RequestAnalysisHostsSingleSized
github.com/minio/warp/pkg/aggregate/requests.go:217 +0x1e8
goroutine 55245 [semacquire]:
sync.runtime_Semacquire(0x0)
runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc09f200000)
sync/waitgroup.go:130 +0x71
github.com/minio/warp/pkg/aggregate.RequestAnalysisHostsSingleSized({0xc09f200000, 0x7960ea, 0x7987f8})
github.com/minio/warp/pkg/aggregate/requests.go:230 +0xa5
github.com/minio/warp/pkg/aggregate.RequestAnalysisSingleSized({0xc09f200000, 0x7960ea, 0x7987f8}, 0x1)
github.com/minio/warp/pkg/aggregate/requests.go:204 +0x105
github.com/minio/warp/pkg/aggregate.Aggregate.func2(0x1)
github.com/minio/warp/pkg/aggregate/aggregate.go:228 +0x792
created by github.com/minio/warp/pkg/aggregate.Aggregate
github.com/minio/warp/pkg/aggregate/aggregate.go:166 +0x84c
`
16GB memory the benchmark vm has and swap turned off.
Shouldn't be enough?
You need a good amount of memory @Badb0yBadb0y to process the warp results - this is simply running out of memory that's all. Run this on a beefier machine.
@Badb0yBadb0y You can use warp analyze warp-remote-2021-12-01[023543]-yh78.csv.zst to analyze the data on a machine with more RAM. The data is saved in the file.
Or you can use --analyze.limit=N to analyze only N operations, combine with --analyze.offset=M to skip first M operations.