Prefetching and total threads
hannahhoward opened this issue · comments
Capturing an issue from @magik6k --
It seems to be doing some kind of prefetching, to the point where occasionally I will see the process die with the "runtime: program exceeds 10000-thread limit" error
I have tuned down the
lassie.WithConcurrentSPRetrievals(50),
lassie.WithBitswapConcurrency(50),
settings from 1000, but I'm still seeing super aggressive prefetching when e.g. trying to open wikipedia in a browser
If you still have my instance around, opening http://127.0.0.1:8080/ipfs/bafybeicvwe7hv3uap425wgd57tda46wfpowqensytht55pmkchnz2xvoxa/wiki/ should do it.
Looking at the summary goroutine dumps I'm seeing:
goroutine profile: total 11722
6000 @ 0x11fd2d6 0x120ddfe 0x2b319db 0x1233441
# 0x2b319da github.com/filecoin-project/lassie/pkg/retriever/bitswaphelpers.(*PreloadCachingStorage).run.func2+0xda /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/bitswaphelpers/preloadcachingstorage.go:414
794 @ 0x11fd2d6 0x120ddfe 0x20df9cc 0x1233441
# 0x20df9cb github.com/ipfs/go-graphsync/taskqueue.(*WorkerTaskQueue).worker+0x18b /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.14.7/taskqueue/taskqueue.go:121
Another dump:
goroutine profile: total 11186
4257 @ 0x11c32dc 0x2db906c 0x2dc84e5 0x2dc8399 0x2dc6fd7 0x1d8b29a 0x1d7eb43 0x1d8137d 0x1d81215 0x1d7f967 0x1d8112c 0x2df8ce5 0x2df8c7c 0x2e04307 0x2b376c5 0x2b2ba7a 0x2b3713a 0x2b369a8 0x1233441
# 0x2db906b github.com/mattn/go-sqlite3._Cfunc_sqlite3_exec+0x4b _cgo_gotypes.go:857
# 0x2dc84e4 github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5.1+0x64 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
# 0x2dc8398 github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5+0x38 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
# 0x2dc6fd6 github.com/mattn/go-sqlite3.(*SQLiteDriver).Open+0x3b16 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1738
# 0x1d8b299 database/sql.dsnConnector.Connect+0x39 /usr/lib/go/src/database/sql/sql.go:758
# 0x1d7eb42 database/sql.(*DB).conn+0x762 /usr/lib/go/src/database/sql/sql.go:1387
# 0x1d8137c database/sql.(*DB).query+0x5c /usr/lib/go/src/database/sql/sql.go:1721
# 0x1d81214 database/sql.(*DB).QueryContext.func1+0x54 /usr/lib/go/src/database/sql/sql.go:1704
# 0x1d7f966 database/sql.(*DB).retry+0x46 /usr/lib/go/src/database/sql/sql.go:1538
# 0x1d8112b database/sql.(*DB).QueryContext+0xcb /usr/lib/go/src/database/sql/sql.go:1703
# 0x2df8ce4 database/sql.(*DB).Query+0xc4 /usr/lib/go/src/database/sql/sql.go:1717
# 0x2df8c7b github.com/lotus-web3/ribs/rbdeal.(*ribsDB).GetRetrievalCandidates+0x5b /home/magik6k/github.com/lotus_web3/ribs/rbdeal/deal_db.go:1191
# 0x2e04306 github.com/lotus-web3/ribs/rbdeal.(*retrievalProvider).FindCandidatesAsync+0x1e6 /home/magik6k/github.com/lotus_web3/ribs/rbdeal/retr_provider.go:87
# 0x2b376c4 github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates.func2+0x84 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:70
# 0x2b2ba79 github.com/filecoin-project/lassie/pkg/internal/candidatebuffer.(*CandidateBuffer).BufferStream+0x139 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/internal/candidatebuffer/candidatebuffer.go:63
# 0x2b37139 github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates+0x3f9 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:66
# 0x2b369a7 github.com/filecoin-project/lassie/pkg/retriever/combinators.RetrieverWithCandidateFinder.Retrieve.func1+0x1c7 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/combinators/retrieverwithcandidatefinder.go:31
1791 @ 0x11c32dc 0x2db758c 0x2dcd966 0x2dcc765 0x2dcc4f3 0x1d87ada 0x1d8798f 0x1d89eec 0x1d8790f 0x2df8de7 0x2e04307 0x2b376c5 0x2b2ba7a 0x2b3713a 0x2b369a8 0x1233441
# 0x2db758b github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal+0x4b _cgo_gotypes.go:367
# 0x2dcd965 github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked.func1+0x45 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:2186
# 0x2dcc764 github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked+0x44 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:2186
# 0x2dcc4f2 github.com/mattn/go-sqlite3.(*SQLiteRows).Next+0x2f2 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:2163
# 0x1d87ad9 database/sql.(*Rows).nextLocked+0x119 /usr/lib/go/src/database/sql/sql.go:2974
# 0x1d8798e database/sql.(*Rows).Next.func1+0x2e /usr/lib/go/src/database/sql/sql.go:2952
# 0x1d89eeb database/sql.withLock+0x8b /usr/lib/go/src/database/sql/sql.go:3405
# 0x1d8790e database/sql.(*Rows).Next+0x6e /usr/lib/go/src/database/sql/sql.go:2951
# 0x2df8de6 github.com/lotus-web3/ribs/rbdeal.(*ribsDB).GetRetrievalCandidates+0x1c6 /home/magik6k/github.com/lotus_web3/ribs/rbdeal/deal_db.go:1201
# 0x2e04306 github.com/lotus-web3/ribs/rbdeal.(*retrievalProvider).FindCandidatesAsync+0x1e6 /home/magik6k/github.com/lotus_web3/ribs/rbdeal/retr_provider.go:87
# 0x2b376c4 github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates.func2+0x84 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:70
# 0x2b2ba79 github.com/filecoin-project/lassie/pkg/internal/candidatebuffer.(*CandidateBuffer).BufferStream+0x139 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/internal/candidatebuffer/candidatebuffer.go:63
# 0x2b37139 github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates+0x3f9 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:66
# 0x2b369a7 github.com/filecoin-project/lassie/pkg/retriever/combinators.RetrieverWithCandidateFinder.Retrieve.func1+0x1c7 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/combinators/retrieverwithcandidatefinder.go:31
800 @ 0x11fd2d6 0x120ddfe 0x20df9cc 0x1233441
# 0x20df9cb github.com/ipfs/go-graphsync/taskqueue.(*WorkerTaskQueue).worker+0x18b /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.14.7/taskqueue/taskqueue.go:121
Another:
goroutine profile: total 9077
1311 @ 0x11c32dc 0x2db906c 0x2dc84e5 0x2dc8399 0x2dc6fd7 0x1d8b29a 0x1d7eb43 0x1d8137d 0x1d81215 0x1d7f967 0x1d8112c 0x2df8ce5 0x2df8c7c 0x2e04307 0x2b376c5 0x2b2ba7a 0x2b3713a 0x2b369a8 0x1233441
# 0x2db906b github.com/mattn/go-sqlite3._Cfunc_sqlite3_exec+0x4b _cgo_gotypes.go:857
# 0x2dc84e4 github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5.1+0x64 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
# 0x2dc8398 github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5+0x38 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
# 0x2dc6fd6 github.com/mattn/go-sqlite3.(*SQLiteDriver).Open+0x3b16 /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1738
# 0x1d8b299 database/sql.dsnConnector.Connect+0x39 /usr/lib/go/src/database/sql/sql.go:758
# 0x1d7eb42 database/sql.(*DB).conn+0x762 /usr/lib/go/src/database/sql/sql.go:1387
# 0x1d8137c database/sql.(*DB).query+0x5c /usr/lib/go/src/database/sql/sql.go:1721
# 0x1d81214 database/sql.(*DB).QueryContext.func1+0x54 /usr/lib/go/src/database/sql/sql.go:1704
# 0x1d7f966 database/sql.(*DB).retry+0x46 /usr/lib/go/src/database/sql/sql.go:1538
# 0x1d8112b database/sql.(*DB).QueryContext+0xcb /usr/lib/go/src/database/sql/sql.go:1703
# 0x2df8ce4 database/sql.(*DB).Query+0xc4 /usr/lib/go/src/database/sql/sql.go:1717
# 0x2df8c7b github.com/lotus-web3/ribs/rbdeal.(*ribsDB).GetRetrievalCandidates+0x5b /home/magik6k/github.com/lotus_web3/ribs/rbdeal/deal_db.go:1191
# 0x2e04306 github.com/lotus-web3/ribs/rbdeal.(*retrievalProvider).FindCandidatesAsync+0x1e6 /home/magik6k/github.com/lotus_web3/ribs/rbdeal/retr_provider.go:87
# 0x2b376c4 github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates.func2+0x84 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:70
# 0x2b2ba79 github.com/filecoin-project/lassie/pkg/internal/candidatebuffer.(*CandidateBuffer).BufferStream+0x139 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/internal/candidatebuffer/candidatebuffer.go:63
# 0x2b37139 github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates+0x3f9 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:66
# 0x2b369a7 github.com/filecoin-project/lassie/pkg/retriever/combinators.RetrieverWithCandidateFinder.Retrieve.func1+0x1c7 /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/combinators/retrieverwithcandidatefinder.go:31
950 @ 0x11fd2d6 0x120ddfe 0x18fe8f6 0x1233441
# 0x18fe8f5 github.com/ipfs/boxo/bitswap/client/internal/messagequeue.(*MessageQueue).runQueue+0x215 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/boxo@v0.10.3-0.20230629151409-cfcbdbce297a/bitswap/client/internal/messagequeue/messagequeue.go:424
893 @ 0x11fd2d6 0x120ddfe 0x1bdecff 0x1bc160b 0x1233441
# 0x1bdecfe github.com/quic-go/quic-go.(*sendQueue).Run+0xfe /home/magik6k/.opt/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/send_queue.go:67
# 0x1bc160a github.com/quic-go/quic-go.(*connection).run.func2+0x2a /home/magik6k/.opt/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/connection.go:533
Logs:
https://gist.github.com/magik6k/a5b217a5d00d00a86cbcfffe063ad0c5
