[BUG] Invalid Merkle Root issue makes the relayer to start syncing from scratch
shekohex opened this issue · comments
Logs
2023-05-09T15:05:50.067426Z DEBUG webb_probe: kind: tx_queue, ty: "EVM", chain_id: 1287, pending: true, tx_hash: 0x7896…ea21
at crates/tx-queue/src/evm/evm_tx_queue.rs:147
in webb_relayer_tx_queue::evm::evm_tx_queue::run with chain: 1287
2023-05-09T15:05:50.067479Z INFO webb_relayer_tx_queue::evm::evm_tx_queue: Tx 0x7896f6637c987c021e923c3705c79c412a70a3eef7293c617e2c4e9c0a80ea21 is submitted and pending!
at crates/tx-queue/src/evm/evm_tx_queue.rs:164
in webb_relayer_tx_queue::evm::evm_tx_queue::run with chain: 1287
2023-05-09T15:05:50.129906Z DEBUG webb_probe: kind: encrypted_outputs_store, encrypted_output_index: 44, encrypted_output: 0xb62839798f238fe43f4b9cc4e3b8afe1d980abf9f362712fc2477c03578edf5491280f05158dde60a157d0fe41be78369968a0d50298f450e8775d8b61c91057c69a38d1d806bf1ccb1bf261412b4eca90d7d1cfe85d07705ea37e6393772d0ea8ef10cfbfcc437e6c1ea6ef15b5bfa76dcb3fd037dec3bb47ec7389d6e035aca610199f768c154b22db639d27b30b300e0c43687f5eae7895613b6db51cb322ef6192c68988e113, chain_id: 11155111, block_number: 3435712
at event-watchers/evm/src/vanchor/vanchor_encrypted_outputs_handler.rs:87
in webb_ew_evm::vanchor::vanchor_encrypted_outputs_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.131294Z DEBUG webb_probe: kind: mt_insert, leaf_index: 44, leaf: 0x0bd1…6461, chain_id: 11155111, block_number: 3435712
at event-watchers/evm/src/vanchor/vanchor_deposit_handler.rs:91
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.131330Z DEBUG webb_ew_evm::vanchor::vanchor_deposit_handler: VAnchor new leaf event, event: NewCommitmentFilter { commitment: 5346275865311793971825485342020976448341637212764711510010043809747699393633, sub_tree_index: 0, leaf_index: 44, encrypted_output: Bytes(0xb62839798f238fe43f4b9cc4e3b8afe1d980abf9f362712fc2477c03578edf5491280f05158dde60a157d0fe41be78369968a0d50298f450e8775d8b61c91057c69a38d1d806bf1ccb1bf261412b4eca90d7d1cfe85d07705ea37e6393772d0ea8ef10cfbfcc437e6c1ea6ef15b5bfa76dcb3fd037dec3bb47ec7389d6e035aca610199f768c154b22db639d27b30b300e0c43687f5eae7895613b6db51cb322ef6192c68988e113) }
at event-watchers/evm/src/vanchor/vanchor_deposit_handler.rs:109
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.131387Z DEBUG webb_ew_evm::vanchor::vanchor_deposit_handler: VAnchor new leaf index does not satisfy the condition, skipping proposal., leaf_index: 44, is_even_index: true
at event-watchers/evm/src/vanchor/vanchor_deposit_handler.rs:115
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.588190Z DEBUG webb_probe: kind: leaves_store, leaf_index: 44, leaf: 0bd1e2893a96b6f023cf6c0f593d989c32f51eac416e781b54d363df54fa6461, chain_id: 11155111, block_number: 3435712
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:185
in webb_ew_evm::vanchor::vanchor_leaves_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.829139Z DEBUG webb_probe: kind: mt_insert, leaf_index: 45, leaf: 0x12cf…d2c4, chain_id: 11155111, block_number: 3435712
at event-watchers/evm/src/vanchor/vanchor_deposit_handler.rs:91
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.829191Z DEBUG webb_ew_evm::vanchor::vanchor_deposit_handler: VAnchor new leaf event, event: NewCommitmentFilter { commitment: 8508960608189368666779912500316077589967167431914706760138338323088399258308, sub_tree_index: 0, leaf_index: 45, encrypted_output: Bytes(0x2740fae754dd53a386f14ac218d549b0c362a494a6faa814e8740d2c21f91635889acc310bf970b236aeb091ac16bef65f6184232be89b687da66e8265b9a1aa2a69f00c68af4e9ccf13bda093b5005971dd35ea2540aeb420bd8fbf75497a4d72aca10582a0c767ec72080ee037e8df6289e0444a9f771b025179fabcb49f3a20aaecc1419dc14ad03f649c3c451f3f1bc6310a795ca6f900a866f25054e053b57eeccc1e326fbe) }
at event-watchers/evm/src/vanchor/vanchor_deposit_handler.rs:109
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.830475Z DEBUG webb_probe: kind: encrypted_outputs_store, encrypted_output_index: 45, encrypted_output: 0x2740fae754dd53a386f14ac218d549b0c362a494a6faa814e8740d2c21f91635889acc310bf970b236aeb091ac16bef65f6184232be89b687da66e8265b9a1aa2a69f00c68af4e9ccf13bda093b5005971dd35ea2540aeb420bd8fbf75497a4d72aca10582a0c767ec72080ee037e8df6289e0444a9f771b025179fabcb49f3a20aaecc1419dc14ad03f649c3c451f3f1bc6310a795ca6f900a866f25054e053b57eeccc1e326fbe, chain_id: 11155111, block_number: 3435712
at event-watchers/evm/src/vanchor/vanchor_encrypted_outputs_handler.rs:87
in webb_ew_evm::vanchor::vanchor_encrypted_outputs_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:50.831725Z WARN webb_ew_evm::vanchor::vanchor_leaves_handler: Invalid merkle root .. Restarting
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:111
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.060490Z DEBUG webb_probe: kind: retry, should_retry: true, error: rate limit exceeded for eth_getlogs or other heavy rpc methods.
at crates/relayer-context/src/ethers_retry_policy.rs:86
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(1287), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.071102Z WARN webb_ew_evm::vanchor::vanchor_leaves_handler: Clear merkle tree...!
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:124
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.071139Z WARN webb_ew_evm::vanchor::vanchor_leaves_handler: Clear local leaves cache...!
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:127
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.072034Z WARN webb_ew_evm::vanchor::vanchor_leaves_handler: Clear last deposit block number...!
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:130
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.072129Z WARN webb_ew_evm::vanchor::vanchor_leaves_handler: Reset block number to contract deployed at...!
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:138
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.084021Z DEBUG webb_probe: kind: retry, should_retry: true, error: rate limit exceeded for eth_getlogs or other heavy rpc methods.
at crates/relayer-context/src/ethers_retry_policy.rs:86
in webb_relayer_tx_queue::evm::evm_tx_queue::run with chain: 1287
2023-05-09T15:05:51.101529Z DEBUG webb_probe: kind: retry, should_retry: true, error: rate limit exceeded for eth_getlogs or other heavy rpc methods.
at crates/relayer-context/src/ethers_retry_policy.rs:86
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(1287), address: 0x441f…d69d, tag: Signature Bridge Watcher
2023-05-09T15:05:51.309122Z DEBUG webb_proposal_signing_backends::mocked: Signaling Signature Bridge to execute proposal, bridge_key: Bridge(Evm(5)), proposal: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000000005c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7"
at crates/proposal-signing-backends/src/mocked.rs:73
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.309173Z DEBUG webb_probe: kind: signing_backend, backend: "Mocked", signal_bridge: Bridge(Evm(5)), data: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000000005c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7", signature: "e2c223554d787203f77dfab357895c0b4e91fcef8e95a228817a1787537f07ad0bd32bfa04dff0a86e8b41cbab64735b8c91506cffb96c2d877dcd39251532f51c"
at crates/proposal-signing-backends/src/mocked.rs:79
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.311258Z DEBUG webb_proposal_signing_backends::mocked: Signaling Signature Bridge to execute proposal, bridge_key: Bridge(Evm(43113)), proposal: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d01000000a869c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7"
at crates/proposal-signing-backends/src/mocked.rs:73
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.311296Z DEBUG webb_probe: kind: signing_backend, backend: "Mocked", signal_bridge: Bridge(Evm(43113)), data: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d01000000a869c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7", signature: "aeab0fbccd914a58fe6ed2ec4115c46159fc284e3462fadcc6d8d52bf687cce54f4c56a124d5dd23511747d37a0d18e3f8cc3a3bc0239d58f4f1b3c7d8990e331c"
at crates/proposal-signing-backends/src/mocked.rs:79
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.314045Z DEBUG webb_proposal_signing_backends::mocked: Signaling Signature Bridge to execute proposal, bridge_key: Bridge(Evm(534353)), proposal: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000082751c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7"
at crates/proposal-signing-backends/src/mocked.rs:73
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.314080Z DEBUG webb_probe: kind: signing_backend, backend: "Mocked", signal_bridge: Bridge(Evm(534353)), data: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000082751c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7", signature: "1f49bb89b19769008e4460d01fab37eafdd5dc89ea5c18532283360653952e225552c01e74c10ace15f1cf3ff1892e3969d417d1ce63ac85d376568d2dd09a3e1c"
at crates/proposal-signing-backends/src/mocked.rs:79
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.315773Z DEBUG webb_proposal_signing_backends::mocked: Signaling Signature Bridge to execute proposal, bridge_key: Bridge(Evm(80001)), proposal: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000013881c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7"
at crates/proposal-signing-backends/src/mocked.rs:73
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.315810Z DEBUG webb_probe: kind: signing_backend, backend: "Mocked", signal_bridge: Bridge(Evm(80001)), data: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000013881c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7", signature: "1d9e2cb2ba05e571d5ab02727b28ae65c09bfb173405ba36a47808f1793cc3dc0bbf8a242591817d1643db8033033c8254cc1569a6eaa13794abab7fb90885481c"
at crates/proposal-signing-backends/src/mocked.rs:79
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.317697Z DEBUG webb_proposal_signing_backends::mocked: Signaling Signature Bridge to execute proposal, bridge_key: Bridge(Evm(1287)), proposal: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000000507c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7"
at crates/proposal-signing-backends/src/mocked.rs:73
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.317731Z DEBUG webb_probe: kind: signing_backend, backend: "Mocked", signal_bridge: Bridge(Evm(1287)), data: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000000507c1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7", signature: "4ac7bb00a407150c21c7a80b94b4702868f3cdb22de8569f7e908625edc47e8265c6adb807468f1f4af4c20ef8945c63c0d1f450ccd41b8d06e212d4fdbad8411c"
at crates/proposal-signing-backends/src/mocked.rs:79
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.319282Z DEBUG webb_proposal_signing_backends::mocked: Signaling Signature Bridge to execute proposal, bridge_key: Bridge(Evm(421613)), proposal: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000066eedc1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7"
at crates/proposal-signing-backends/src/mocked.rs:73
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.319412Z DEBUG webb_probe: kind: signing_backend, backend: "Mocked", signal_bridge: Bridge(Evm(421613)), data: "00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000066eedc1922f9e0000002d113823dae184f88823ad52defb02c331d9fada09ec36dbbdce81ebd1366d213d00000000000038e7aa90c77f86747fab355eecaa0c2e4c3a463d010000aa36a7", signature: "1b0726d835fcd8f966809ce53ea14272b29b3be5c82b25cda2236092a484af852d6fe419be34680ff1e0dc835664ca468ed74d1c92ec41f25f25a2a5325e684a1b"
at crates/proposal-signing-backends/src/mocked.rs:79
in webb_ew_evm::vanchor::vanchor_deposit_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.323128Z DEBUG webb_probe: kind: leaves_store, leaf_index: 45, leaf: 12cfe6aa3d104b56b1dc32771e34e8a4c1fe9b093ef8a9a6829336f5a841d2c4, chain_id: 11155111, block_number: 3435712
at event-watchers/evm/src/vanchor/vanchor_leaves_handler.rs:185
in webb_ew_evm::vanchor::vanchor_leaves_handler::handle_event
in webb_event_watcher_traits::evm::event_watcher::run with chain_id: Ok(11155111), address: 0x38e7…463d, tag: VAnchor Contract Watcher
2023-05-09T15:05:51.340216Z DEBUG webb_probe: kind: retry, should_retry: true, error: rate limit exceeded for eth_getlogs or other heavy rpc methods.
at crates/relayer-context/src/ethers_retry_policy.rs:86
in webb_ew_evm::signature_bridge_watcher::execute_proposal_with_signature
in webb_ew_evm::signature_bridge_watcher::handle_cmd
in webb_event_watcher_traits::evm::bridge_watcher::run with chain_id: Ok(1287), address: 0x441f…d69d, tag: Signature Bridge Watcher
2023-05-09T15:05:51.345767Z DEBUG webb_probe: kind: retry, should_retry: true, error: too many requests
at crates/relayer-context/src/ethers_retry_policy.rs:86
in webb_ew_evm::signature_bridge_watcher::execute_proposal_with_signature
in webb_ew_evm::signature_bridge_watcher::handle_cmd
in webb_event_watcher_traits::evm::bridge_watcher::run with chain_id: Ok(80001), address: 0x441f…d69d, tag: Signature Bridge Watcher
It should resync the relayer from scratch to construct the Merkle tree again.
But we should have a validation check on the deposit handler which will send AnchorUpdate proposals for signing only if nonce = on_chain_nonce + 1.
Currently this just keep looping endless, and it keeps failing. I would argue that we should revert/disable this feature for now as it looks buggy.