Number of digest items must match that calculated
ehsan6sha opened this issue · comments
Is there an existing issue?
- I have searched the existing issues
Experiencing problems? Have you tried our Stack Exchange first?
- This is not a support question.
Description of bug
I updated the runtime code by following these instructions: https://docs.substrate.io/tutorials/build-a-blockchain/upgrade-a-running-network/
(functionland/sugarfunge-node@b073df0)
And now when syncing from 0, the node gives an error on the block that update happened:
2024-05-25 14:21:54 Substrate Node
2024-05-25 14:21:54 ✌️ version 0.1.0-fab981dafd8
2024-05-25 14:21:54 ❤️ by SugarFunge Developers, 2017-2024
2024-05-25 14:21:54 📋 Chain specification: Functionyard
2024-05-25 14:21:54 🏷 Node name: FulaNode
2024-05-25 14:21:54 👤 Role: AUTHORITY
2024-05-25 14:21:54 💾 Database: RocksDb at /uniondrive/chain/chains/functionyard/db/full
2024-05-25 14:21:54 ⛓ Native runtime: sugarfunge-node-101 (sugarfunge-node-1.tx1.au1)
2024-05-25 14:23:14 Using default protocol ID "sup" because none is configured in the chain specs
2024-05-25 14:23:14 🏷 Local node identity is: 12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3
egXp6Jg6WC2Nkkrd
2024-05-25 14:23:14 💻 Operating system: linux
2024-05-25 14:23:14 💻 CPU architecture: aarch64
2024-05-25 14:23:14 💻 Target environment: gnu
2024-05-25 14:23:14 💻 Memory: 7810MB
2024-05-25 14:23:14 💻 Kernel: 6.6.20+rpt-rpi-v8
2024-05-25 14:23:14 💻 Linux distribution: Ubuntu 24.04 LTS
2024-05-25 14:23:14 💻 Virtual machine: no
2024-05-25 14:23:14 📦 Highest known block at #531910
2024-05-25 14:23:14 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-05-25 14:23:14 Running JSON-RPC server: addr=0.0.0.0:9945, allowed origins=["*"]
Connection to 127.0.0.1 9945 port [tcp/*] succeeded!
2024-05-25 14:23:14 Accepting new connection 1/100
2024-05-25 14:23:14 Accepting new connection 1/100
[2024-05-25T14:23:14Z INFO actix_server::builder] starting 4 workers
[2024-05-25T14:23:14Z INFO actix_server::server] Actix runtime found; starting in Actix runtime
2024-05-25 14:23:15 🔍 Discovered new external address for our node: /ip4/86.242.11.169/tcp/30335/p2p/12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3egXp6Jg6WC2Nkkrd
Connection to 127.0.0.1 4000 port [tcp/*] succeeded!
Connection to 127.0.0.1 5001 port [tcp/*] succeeded!
[2024-05-25T14:23:16Z INFO actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.013675
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:19 panicked at /home/ubuntu/.cargo/git/checkouts/substrate-7e08433d4c370a21/5e49f6e/frame/executive/src/lib.rs:588:9:
assertion `left == right` failed: Number of digest items must match that calculated.
left: 2
right: 1
2024-05-25 14:23:19 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
0: 0x219d0 - <unknown>!rust_begin_unwind
1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:19 💔 Error importing block 0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
0: 0x219d0 - <unknown>!rust_begin_unwind
1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:19 ⚙️ Syncing, target=#694235 (1 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 82.0kiB/s ⬆ 1.3kiB/s
2024-05-25 14:23:20 panicked at /home/ubuntu/.cargo/git/checkouts/substrate-7e08433d4c370a21/5e49f6e/frame/executive/src/lib.rs:588:9:
assertion `left == right` failed: Number of digest items must match that calculated.
left: 2
right: 1
2024-05-25 14:23:20 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
0: 0x219d0 - <unknown>!rust_begin_unwind
1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:20 💔 Error importing block 0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
0: 0x219d0 - <unknown>!rust_begin_unwind
1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
5: 0xb1d2e - <unknown>!Core_execute_block
[2024-05-25T14:23:21Z INFO actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001207
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:24 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 80.4kiB/s ⬆ 0.2kiB/s
[2024-05-25T14:23:26Z INFO actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.003389
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:29 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 21 B/s ⬆ 35 B/s
2024-05-25 14:23:30 🔍 Discovered new external address for our node: /ip6/2a01:cb00:12a6:6500:2664:12f7:d79d:7bd2/tcp/30335/p2p/12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3egXp6Jg6WC2Nkkrd
[2024-05-25T14:23:31Z INFO actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001223
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:34 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 1.9kiB/s ⬆ 0.9kiB/s
[2024-05-25T14:23:36Z INFO actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 54 "-" "curl/8.5.0" 0.001243
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:39 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 0.5kiB/s ⬆ 0.2kiB/s
[2024-05-25T14:23:41Z INFO actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001217
Waiting for Node API to have at least 3 peers...
Any ideas of how to troubleshoot further?
https://functionyard.fula.network/#/explorer/query/0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db
Steps to reproduce
- upgrade a running node
- sync from scratch
However, strangely it happens on some nodes and not all
I tried this one by setting state cache size to 0 but it did not solve the issue:
paritytech/substrate#9697
Did you increment the spec version?
Did you increment the spec version?
Yes, both spec_version and EXISTENTIAL_DEPOSIT
https://github.com/functionland/sugarfunge-node/blob/main/runtime/src/lib.rs
any ideas on what is causing it or how to troubleshoot further?