sync issue 13.01.2024
alexqrid opened this issue · comments
I'm running the latest tag v1.12 and have sync issue:
Periodically getting Environment mapsize limit reached on various buckets
Execution failed block=35216884 hash=0xa56f4d7fc69967dcc90883807b3d9778fa42ebf57fe84c62f0f7ca5afecd7232 err="writing changesets for block 35216884 failed: in AppendDup: bucket=AccountChangeSet, mdbx_cursor_put: MDBX_MAP_FULL: Environment mapsize limit reached"
[INFO] [01-13|20:40:28.861] UnwindTo block=35216883 bad_block_hash=0xa56f4d7fc69967dcc90883807b3d9778fa42ebf57fe84c62f0f7ca5afecd7232
[EROR] [01-13|20:40:30.881] Staged Sync err="[7/15 Execution] batch commit: table: PlainState, mdbx_cursor_open: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid, stack: [kv_mdbx.go:1058 kv_mdbx.go:1036 collector.go:147 mapmutation.go:236 mapmutation.go:251 stage_execute.go:521 default_stages.go:101 sync.go:358 sync.go:260 stageloop.go:177 stageloop.go:92 asm_amd64.s:1598]"
Another log^
[WARN] [01-13|22:35:47.699] [7/15 Execution] Execution failed block=35215547 hash=0xa2a867e352c7cd005f678d32fd855b609a48c0e44fa799a8fd831b0f275a2a7a err="writing receipts for block 35215547: bucket: TransactionLog, mdbx_cursor_put: MDBX_MAP_FULL: Environment mapsize limit reached"
[INFO] [01-13|22:35:47.699] UnwindTo block=35215546 bad_block_hash=0xa2a867e352c7cd005f678d32fd855b609a48c0e44fa799a8fd831b0f275a2a7a
[EROR] [01-13|22:35:49.545] Staged Sync err="[7/15 Execution] batch commit: table: PlainState, mdbx_cursor_open: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid, stack: [kv_mdbx.go:1058 kv_mdbx.go:1036 collector.go:147 mapmutation.go:236 mapmutation.go:251 stage_execute.go:521 default_stages.go:101 sync.go:358 sync.go:260 stageloop.go:177 stageloop.go:92 asm_amd64.s:1598]"
Node startup script:
erigon
--metrics
--sentry.drop-useless-peers=true
--metrics.addr 0.0.0.0
--metrics.port 9217
--datadir /home/bsc/.new_erigon_data/
--txpool.globalqueue 40000
--trace.maxtraces 100500
--txpool.accountslots 16
--txpool.globalslots 10000
--txpool.lifetime 4h0m0s
--batchSize 1024M
--private.api.ratelimit 100500
--db.pagesize=16kb
--etl.bufferSize 4096MB
--maxpeers 50
--healthcheck
--snapshots=false
--snap.stop=true
--networkid 56
--private.api.addr '127.0.0.1:10066'
--p2p.protocol=66
--p2p.allowed-ports 30075,30076,30077
--port 30078
--chain bsc
--torrent.download.rate=512mb
--torrent.upload.rate=10mb
--torrent.port=42013
Full log from node restart
[INFO] [01-13|23:05:50.548] New txs subscriber joined
[INFO] [01-13|23:05:57.822] [2/15 Headers] Processed highest inserted=35221311 age=1s
[INFO] [01-13|23:05:58.257] [5/15 Bodies] Processing bodies... from=35211385 to=35221311
[INFO] [01-13|23:06:18.257] [5/15 Bodies] Downloading block bodies block_num=35212971 delivery/sec=5.9MB wasted/sec=0B remaining=8339 delivered=1820 cache=15.2MB alloc=3.4GB sys=3.7GB
[INFO] [01-13|23:06:38.257] [5/15 Bodies] Downloading block bodies block_num=35215409 delivery/sec=7.1MB wasted/sec=0B remaining=5901 delivered=4082 cache=3.3MB alloc=4.2GB sys=4.5GB
[INFO] [01-13|23:06:58.258] [5/15 Bodies] Downloading block bodies block_num=35217662 delivery/sec=6.6MB wasted/sec=0B remaining=3648 delivered=6679 cache=19.3MB alloc=2.7GB sys=5.0GB
[INFO] [01-13|23:07:18.258] [5/15 Bodies] Downloading block bodies block_num=35220217 delivery/sec=5.3MB wasted/sec=0B remaining=1109 delivered=9040 cache=8.5MB alloc=3.4GB sys=5.0GB
[INFO] [01-13|23:07:24.316] [5/15 Bodies] Processed highest=35221311
[INFO] [01-13|23:07:24.316] [5/15 Bodies] DONE in=1m26.059137337s
[INFO] [01-13|23:07:24.316] [6/15 Senders] Started from=35211385 to=35221311
[INFO] [01-13|23:07:28.017] [7/15 Execution] Blocks execution from=35211385 to=35221311
[INFO] [01-13|23:07:41.291] [p2p] GoodPeers eth66=40
[INFO] [01-13|23:07:48.043] [7/15 Execution] Executed blocks number=35211753 blk/s=18.4 tx/s=3216.3 Mgas/s=314.5 gasState=0.00 batch=12.2MB alloc=3.3GB sys=5.4GB
[INFO] [01-13|23:08:08.031] [7/15 Execution] Executed blocks number=35212161 blk/s=20.4 tx/s=3279.5 Mgas/s=349.3 gasState=0.01 batch=27.8MB alloc=3.8GB sys=5.4GB
[INFO] [01-13|23:08:28.037] [7/15 Execution] Executed blocks number=35212631 blk/s=23.5 tx/s=3488.8 Mgas/s=370.9 gasState=0.01 batch=45.8MB alloc=4.5GB sys=5.4GB
[INFO] [01-13|23:08:48.025] [7/15 Execution] Executed blocks number=35212996 blk/s=18.3 tx/s=3013.3 Mgas/s=342.9 gasState=0.01 batch=60.9MB alloc=2.7GB sys=5.5GB
[INFO] [01-13|23:09:08.046] [7/15 Execution] Executed blocks number=35213409 blk/s=20.6 tx/s=2976.6 Mgas/s=356.9 gasState=0.02 batch=77.2MB alloc=3.2GB sys=5.5GB
[INFO] [01-13|23:09:28.044] [7/15 Execution] Executed blocks number=35213916 blk/s=25.4 tx/s=3494.7 Mgas/s=371.8 gasState=0.02 batch=94.7MB alloc=3.9GB sys=5.5GB
[INFO] [01-13|23:09:41.291] [p2p] GoodPeers eth66=49
[INFO] [01-13|23:09:48.044] [7/15 Execution] Executed blocks number=35214370 blk/s=22.7 tx/s=3077.7 Mgas/s=344.9 gasState=0.02 batch=110.3MB alloc=4.3GB sys=5.6GB
[INFO] [01-13|23:10:08.050] [7/15 Execution] Executed blocks number=35214955 blk/s=29.2 tx/s=3665.4 Mgas/s=373.2 gasState=0.03 batch=128.1MB alloc=5.1GB sys=5.6GB
WARN] [01-13|23:10:15.640] [7/15 Execution] Execution failed block=35215192 hash=0x96365f88e4807b60136b970a912216600fa17eed9a72e80229824e1aaea369c4 err="writing receipts for block 35215192: bucket: TransactionLog, mdbx_cursor_put: MDBX_MAP_FULL: Environment mapsize limit reached"
[INFO] [01-13|23:10:15.640] UnwindTo block=35215191 bad_block_hash=0x96365f88e4807b60136b970a912216600fa17eed9a72e80229824e1aaea369c4
EROR] [01-13|23:10:15.802] Staged Sync err="[7/15 Execution] batch commit: table: Code, mdbx_cursor_open: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid, stack: [kv_mdbx.go:1058 kv_mdbx.go:1043 collector.go:147 mapmutation.go:236 mapmutation.go:251 stage_execute.go:521 default_stages.go:101 sync.go:358 sync.go:260 stageloop.go:177 stageloop.go:92 asm_amd64.s:1598]"
[INFO] [01-13|23:10:16.625] [2/15 Headers] Waiting for headers... from=35211385
[INFO] [01-13|23:10:30.019] [2/15 Headers] Processed highest inserted=35221402 age=1s
[INFO] [01-13|23:10:30.321] [5/15 Bodies] Processing bodies... from=35211385 to=35221402
[INFO] [01-13|23:10:50.321] [5/15 Bodies] Downloading block bodies block_num=35211581 delivery/sec=4.2MB wasted/sec=0B remaining=9820 delivered=1305 cache=71.9MB alloc=5.1GB sys=5.8GB
[INFO] [01-13|23:11:10.560] [5/15 Bodies] Downloading block bodies block_num=35211820 delivery/sec=10.6MB wasted/sec=0B remaining=9581 delivered=4768 cache=247.2MB alloc=3.6GB sys=6.0GB
[INFO] [01-13|23:11:30.322] [5/15 Bodies] Downloading block bodies block_num=35219900 delivery/sec=10.7MB wasted/sec=0B remaining=1502 delivered=9248 cache=6.6MB alloc=4.8GB sys=6.6GB
[INFO] [01-13|23:11:38.900] [5/15 Bodies] Processed highest=35221402
[INFO] [01-13|23:11:38.900] [5/15 Bodies] DONE in=1m8.579162743s
[INFO] [01-13|23:11:38.900] [6/15 Senders] Started from=35211385 to=35221402
[INFO] [01-13|23:11:41.174] [7/15 Execution] Blocks execution from=35211385 to=35221402
[INFO] [01-13|23:11:41.291] [p2p] GoodPeers eth66=38
[INFO] [01-13|23:12:01.185] [7/15 Execution] Executed blocks number=35211798 blk/s=20.6 tx/s=3601.8 Mgas/s=356.4 gasState=0.00 batch=14.0MB alloc=2.7GB sys=6.7GB
[INFO] [01-13|23:12:21.211] [7/15 Execution] Executed blocks number=35212258 blk/s=23.0 tx/s=3675.9 Mgas/s=391.7 gasState=0.01 batch=31.9MB alloc=3.6GB sys=6.7GB
[INFO] [01-13|23:12:41.180] [7/15 Execution] Executed blocks number=35212771 blk/s=25.7 tx/s=3756.8 Mgas/s=398.2 gasState=0.01 batch=51.3MB alloc=4.6GB sys=6.7GB
[INFO] [01-13|23:13:01.211] [7/15 Execution] Executed blocks number=35213138 blk/s=18.3 tx/s=3005.6 Mgas/s=359.4 gasState=0.01 batch=66.2MB alloc=2.9GB sys=6.7GB
[INFO] [01-13|23:13:21.182] [7/15 Execution] Executed blocks number=35213626 blk/s=24.4 tx/s=3477.8 Mgas/s=392.3 gasState=0.02 batch=85.3MB alloc=3.6GB sys=6.7GB
[INFO] [01-13|23:13:41.208] [7/15 Execution] Executed blocks number=35214139 blk/s=25.6 tx/s=3515.4 Mgas/s=383.4 gasState=0.02 batch=102.7MB alloc=4.3GB sys=6.7GB
[INFO] [01-13|23:13:41.291] [p2p] GoodPeers eth66=43
[INFO] [01-13|23:14:01.178] [7/15 Execution] Executed blocks number=35214682 blk/s=27.2 tx/s=3508.4 Mgas/s=375.6 gasState=0.02 batch=120.0MB alloc=5.1GB sys=6.7GB
WARN] [01-13|23:14:16.033] [7/15 Execution] Execution failed block=35215144 hash=0xc31947f07ebc23b66cd46106bd326777f65e6440a0f565f26b8e4ff819aa5f12 err="writing receipts for block 35215144: bucket: TransactionLog, mdbx_cursor_put: MDBX_MAP_FULL: Environment mapsize limit reached"
[INFO] [01-13|23:14:16.033] UnwindTo block=35215143 bad_block_hash=0xc31947f07ebc23b66cd46106bd326777f65e6440a0f565f26b8e4ff819aa5f12
EROR] [01-13|23:14:17.846] Staged Sync err="[7/15 Execution] batch commit: table: PlainState, mdbx_cursor_open: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid, stack: [kv_mdbx.go:1058 kv_mdbx.go:1036 collector.go:147 mapmutation.go:236 mapmutation.go:251 stage_execute.go:521 default_stages.go:101 sync.go:358 sync.go:260 stageloop.go:177 stageloop.go:92 asm_amd64.s:1598]"
[INFO] [01-13|23:14:18.662] [2/15 Headers] Waiting for headers... from=35211385
[INFO] [01-13|23:14:33.416] [2/15 Headers] Processed highest inserted=35221483 age=1s
[INFO] [01-13|23:14:33.599] [5/15 Bodies] Processing bodies... from=35211385 to=35221483
[INFO] [01-13|23:14:53.599] [5/15 Bodies] Downloading block bodies block_num=35212318 delivery/sec=4.2MB wasted/sec=0B remaining=9164 delivered=1287 cache=21.2MB alloc=4.3GB sys=6.7GB
[INFO] [01-13|23:15:13.600] [5/15 Bodies] Downloading block bodies block_num=35215307 delivery/sec=8.6MB wasted/sec=0B remaining=6175 delivered=3996 cache=4.4MB alloc=5.2GB sys=6.7GB
Now the data need more than 8TB, do you have enough disk
Thanks for the response. Yes I've enough space and my datadir is 7.8 Tb at the moment.
make db-tools
./build/bin/mdbx_stat -ef /chaindata
BTW, my datadir is 7.96Tb now. I see you add this flag --snap.stop=true, it means db will not remove the data to snapshots.
I was able to resolve the issue with the following steps:
- Stopped the node
- compile integration tools via
$ make integration - unwinded 100 blocks with the following command
./integration stage_headers --unwind=100 --datadir=<path_to_datadir> --chain bsc - Started the node and wait until it started to download the blocks
- Deleted /txpool directory.
@blxdyx I don't want snapshots to be created, that's why I use that flag, is there a pitfall of not using snapshots?
I was able to resolve the issue with the following steps:
- Stopped the node
- compile integration tools via
$ make integration- unwinded 100 blocks with the following command
./integration stage_headers --unwind=100 --datadir=<path_to_datadir> --chain bsc- Started the node and wait until it started to download the blocks
- Deleted /txpool directory.
@blxdyx I don't want snapshots to be created, that's why I use that flag, is there a pitfall of not using snapshots?
No pitfall, just the chaindata will be more bigger
BTW, i suggest add --txpool.disable this flag, so won't have this directory
Reopen if still have problem