Snap sync stuck at 0% of State Ranges (Phase 1)
coursecarbons opened this issue · comments
Description
After installing the latest version of nethermind, and fter erasing nethermind_db for a full re-sync on a node that stopped validating, I am unable to sync.
After completing the initial part of the sync, it is stuck for 10+ hours at
Snap State Ranges (Phase 1): ( 0.00 %) [ ]
i.e. states are not being sync'ed. I see the CPU working at a very modest pace, very little I/O, and quite a bit of network traffic.
The config file contains
--Sync.SnapSync true
as the only Sync option.
Stopping and restarting nethermind does not seem to help.
Steps to Reproduce
Delete nethermind_db folder, to force a complete re-sync.
Actual behavior
After completing the initial part of the sync, it is stuck for 10+ hours at
Snap State Ranges (Phase 1): ( 0.00 %) [ ]
i.e. states are not being sync'ed.
Expected behavior
Would expect the state sync part of the synchronization to advance in time.
Screenshots
If applicable, please include screenshots to help illustrate the problem.
Desktop (please complete the following information):
Please provide the following information regarding your setup:
- Operating System: Ubuntu
- Version: 22.04.4 LTS
- Installation Method: GitHub release
- Consensus Client: teku/v24.4.0/linux-x86_64/-privatebuild-openjdk64bitservervm-java-17
Additional context
Logs
Typical part of the log, going on for hours now:
Jun 06 16:28:17 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:17 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:18 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:18 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:19 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:19 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:20 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:20 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:21 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:21 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:22 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:22 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:22 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:22 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:23 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:23 | Received ForkChoice: 20031071 (0x466935...089572), Safe: 20031026 (0x63b4bf...d87c94), Finalized: 20030994 (0x2d4708...70a41b)
Jun 06 16:28:23 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:23 | Syncing beacon headers, Request: ForkChoice: 20031071 (0x466935...089572), Safe: 20031026 (0x63b4bf...d87c94), Finalized: 20030994 (0x2d4708...70a41b)
Jun 06 16:28:23 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:23 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:24 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:24 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:25 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:25 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:25 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:25 | Snap - (AccountRange, diff: 0) 8 - Requests Account: 8 | Storage: 0 | Code: 0 | Refresh: 0 - Queues Slots: 0 | Storages: 0 | Codes: 0 | Refresh: 0
Jun 06 16:28:26 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:26 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:27 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:27 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:28 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:28 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:29 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:29 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:30 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:30 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:31 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:31 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:32 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:32 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:33 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:33 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:34 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:34 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:34 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:34 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:34 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:34 | Snap - (Skipped Request, diff: 0) 8 - Requests Account: 8 | Storage: 0 | Code: 0 | Refresh: 0 - Queues Slots: 0 | Storages: 0 | Codes: 0 | Refresh: 0
Jun 06 16:28:35 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:35 | Peers | with best block: 110 | all: 110 | eth66 (2 %), eth67 (12 %), eth68 (86 %) | Active: 8 Snap | Sleeping: 23 Snap | Geth (47 %), Nethermind (25 %), Erigon (16 %), Reth (6 %), Besu (4 %), Unknown (1 %)
Jun 06 16:28:35 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:35 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:36 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:36 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:37 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:37 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:38 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:38 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:39 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:39 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:40 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:40 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:41 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:41 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:42 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:42 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:43 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:43 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:44 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:44 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:44 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:44 | Snap - (Skipped Request, diff: 0) 8 - Requests Account: 8 | Storage: 0 | Code: 0 | Refresh: 0 - Queues Slots: 0 | Storages: 0 | Codes: 0 | Refresh: 0
Jun 06 16:28:45 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:45 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:46 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:46 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:47 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:47 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
Jun 06 16:28:48 Gimly Nethermind.Runner[373877]: 06 Jun 16:28:48 | Snap State Ranges (Phase 1): ( 0.00 %) [ ]
I should have perhaps added that if I initiate a full sync (archive mode), then states start downloading and progressing immediately, so I would rule out any connectivity issues (also because the first part of the sync goes through fine).
Hello, can you double check that your CL is synced?
Indeed the CL was stuck during its syncing phase. I removed its database, re-sync'ed from scratch, and once the CL sync'ed, Nethermind continued on without a glitch. I had not understood that the state syncing phase of Nethermind required the CL to be fully synced.
Thank you so much for your help, very much appreciated.