[BUG] Database error while running insertLogFilterInterval

Question

[BUG] Database error while running insertLogFilterInterval

0xneves opened this issue 3 months ago · comments

0xneves.eth commented 3 months ago

Bug

Version

"@ponder/core": "^0.4.7"

Bug found

...
WARN  historical Block task failed, retrying... [4617] (network=kakarot, error=TypeError: Cannot read properties of undefined (reading 'toString'))
WARN  sync Database error while running insertLogFilterInterval, retrying after 100 milliseconds. Error: Cannot read properties of undefined (reading 'toString')
...

Steps to reproduce

Clone this repo in the ponder branch
Add the public RPC of Kakarot: https://sepolia-rpc.kakarot.org
Run npm run ponder or others alike

Additional Context

There is only one RPC available in the entire world at the moment as the chain is on alpha

χ² · Answer 1 · Sat Apr 20 2024 18:17:00 GMT+0800 (China Standard Time)

This might be helpful, it seems as if the size field in the kakarot blocks is null:

$ cast bl -r https://sepolia-rpc.kakarot.org --json | jq .
{
  "hash": "0x052f14a4fa6271b93dd48996e38d45633a5958d1b2d79d68405114b758828836",
  "parentHash": "0x05003cf2c38f8bafd8b82a4617ff47dc2c2dd8fc5f8792e98139fc753f3d782e",
  "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  "miner": "0xca40796afb5472abaed28907d5ed6fc74c04954a",
  "stateRoot": "0x06eff4a5dc5808ca81ad5702fb4ec1510424cdfc8690167a4e7617511af9e962",
  "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "number": "0x1228",
  "gasUsed": "0x0",
  "gasLimit": "0x1312d00",
  "extraData": "0x",
  "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "timestamp": "0x662384f2",
  "difficulty": "0x0",
  "totalDifficulty": "0x0",
  "sealFields": [],
  "uncles": [],
  "transactions": [],
  "size": null,
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "nonce": "0x0000000000000000",
  "baseFeePerGas": "0x0",
  "withdrawalsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "withdrawals": []
}

Rafael de Souza Sanches · Answer 2 · Fri Apr 26 2024 21:27:32 GMT+0800 (China Standard Time)

This might be helpful, it seems as if the size field in the kakarot blocks is null:

$ cast bl -r https://sepolia-rpc.kakarot.org --json | jq .
{
  "hash": "0x052f14a4fa6271b93dd48996e38d45633a5958d1b2d79d68405114b758828836",
  "parentHash": "0x05003cf2c38f8bafd8b82a4617ff47dc2c2dd8fc5f8792e98139fc753f3d782e",
  "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  "miner": "0xca40796afb5472abaed28907d5ed6fc74c04954a",
  "stateRoot": "0x06eff4a5dc5808ca81ad5702fb4ec1510424cdfc8690167a4e7617511af9e962",
  "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "number": "0x1228",
  "gasUsed": "0x0",
  "gasLimit": "0x1312d00",
  "extraData": "0x",
  "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "timestamp": "0x662384f2",
  "difficulty": "0x0",
  "totalDifficulty": "0x0",
  "sealFields": [],
  "uncles": [],
  "transactions": [],
  "size": null,
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "nonce": "0x0000000000000000",
  "baseFeePerGas": "0x0",
  "withdrawalsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "withdrawals": []
}

excusme, what's is this cast bl command? I'd like to replicate what you did but so that we can check it from here as well. Thx!

χ² · Answer 3 · Fri Apr 26 2024 21:32:58 GMT+0800 (China Standard Time)

@RafaDSan cast is part of the foundry toolkit, which is very helpful for developing/testing Solidity contracts, as well as interacting with RPC nodes in general:

https://github.com/foundry-rs/foundry

Rafael de Souza Sanches · Answer 4 · Fri Apr 26 2024 22:33:17 GMT+0800 (China Standard Time)

@RafaDSan cast is part of the foundry toolkit, which is very helpful for developing/testing Solidity contracts, as well as interacting with RPC nodes in general:

https://github.com/foundry-rs/foundry

thx for the fast answer!

Jay Miller · Answer 5 · Fri Apr 26 2024 22:49:40 GMT+0800 (China Standard Time)

@0xneves Hi thanks for the notes on how to reproduce. I just followed and ran your indexer locally and i'm not getting the bug you encountered last week.

I took a look at the raw RPC data for blocks and it seems like it is correctly reporting size now as non-null. I think @ChiTimesChi is correct about that being the root cause of the bug

Going to close this for now. We will have a more general fix for this issue (non-critical fields sometimes being null in RPC responses) in the near future.

0xneves.eth · Answer 6 · Tue May 07 2024 00:46:07 GMT+0800 (China Standard Time)

@jaylmiller Thank you for the support, I appreciate the help. The blocks are being synced correctly, but something is still out of rail:

The ponder.on event fetcher is not triggering when events are emitted on the blockchain and we don't know why.
When synching the blocks with a contract that has previous events, those won't trigger ponder.on as well.

How can we move further to understand this behavior and fix this? Assuming this problem is from the Kakarot side since other EVMs are working correctly, I wonder if the interface is not matching maybe?

How can I debug for a better understanding of the issue? Can we reopen this issue?

Jay Miller · Answer 7 · Tue May 07 2024 03:03:07 GMT+0800 (China Standard Time)

@0xneves is it possible your chain's RPC may be down or not returning logs data (this would cause no events to be triggered)? if u send the address of the contract you are indexing, we can verify what's happening.

Rafael de Souza Sanches · Answer 8 · Tue May 07 2024 04:28:04 GMT+0800 (China Standard Time)

@0xneves is it possible your chain's RPC may be down or not returning logs data (this would cause no events to be triggered)? if u send the address of the contract you are indexing, we can verify what's happening.

Hello, @jaylmiller ! The contract we have deployed on the network is: 0x80f14F36Eb515C8F254108f8c344D9ef44e3c33e

The network could be quite unstable due to it's alpha phase and at the time I write this message, they are not available.
But we've tested when they were certainly online, with the RPC working
When they get online again, I'll send a message here, is that okay?

Rafael de Souza Sanches · Answer 9 · Wed May 08 2024 01:22:43 GMT+0800 (China Standard Time)

@0xneves is it possible your chain's RPC may be down or not returning logs data (this would cause no events to be triggered)? if u send the address of the contract you are indexing, we can verify what's happening.

Hello, @jaylmiller ! The contract we have deployed on the network is: 0x80f14F36Eb515C8F254108f8c344D9ef44e3c33e

The network could be quite unstable due to it's alpha phase and at the time I write this message, they are not available. But we've tested when they were certainly online, with the RPC working When they get online again, I'll send a message here, is that okay?

the network is online now @jaylmiller @0xOlias
https://sepolia-explorer.kakarot.org/dashboard

Do you guys need any more information?

Jay Miller · Answer 10 · Fri May 10 2024 23:45:02 GMT+0800 (China Standard Time)

There are no logs associated with your address, so you aren't seeing any events. You can use cast to check for logs on your contract:

cast l --address 0x80f14F36Eb515C8F254108f8c344D9ef44e3c33e -r https://sepolia-rpc.kakarot.org

Rafael de Souza Sanches · Answer 11 · Fri May 10 2024 23:49:52 GMT+0800 (China Standard Time)

The network was reset and our contract was erased, and it seems it's not working again. We'll have to wait a bit more to test it again

Rafael de Souza Sanches · Answer 12 · Mon May 13 2024 11:53:23 GMT+0800 (China Standard Time)

New address: 0xB317127b50b22e62637E3c333A585a8ccfd0721D
https://sepolia-explorer.kakarot.org/contracts/0xb317127b50b22e62637e3c333a585a8ccfd0721d

The script cast l --address 0xB317127b50b22e62637E3c333A585a8ccfd0721D -r https://sepolia-rpc.kakarot.org returned:

 address: 0xB317127b50b22e62637E3c333A585a8ccfd0721D
  blockHash: 0x014519dadc66e54ac868dd9c87b85844a310895fb92a74583efe208a5584ef96
  blockNumber: 92
  data: 0x
  logIndex: 0
  removed: false
  topics: [
        0x3aac258f17a492cc33fefd5baf2537a4c279fa275b8f98eae89985d2f2af7109
        0x0000000000000000000000000000000000000000000000000000000000000001
        0x000000000000000000000000a8dfdaafce4e8a69882da636f1cf6af0caf70336
        0x00000000000000000000000011eca31264d6f2356e24089c97cdbf0147a2a447
  ]
  transactionHash: 0x76803c1aa5d2b73e47173f9bccf9648dee047c5595322f55c5a6b0044af80db7
  transactionIndex: 0
- address: 0xB317127b50b22e62637E3c333A585a8ccfd0721D
  blockHash: 0x0564d67415084bbfbad7a1e3f2a077d9a0fb2085481ee863bb6ecc91c2efff81
  blockNumber: 101
  data: 0x
  logIndex: 0
  removed: false
  topics: [
        0x3aac258f17a492cc33fefd5baf2537a4c279fa275b8f98eae89985d2f2af7109
        0x0000000000000000000000000000000000000000000000000000000000000002
        0x00000000000000000000000040379217422dec2f06eddf993ccccd52c90835fd
        0x000000000000000000000000a8dfdaafce4e8a69882da636f1cf6af0caf70336
  ]
  transactionHash: 0x682e4f96423d2d2ae89996472cfaaa323eef77496ce1ee51eb8ae93b7ac6b4f6
  transactionIndex: 0

Which I suppose are the two events emited by the contract, right?

But why exactly these events are not being find by the ponder application as we can see in the print below, where the "kakarot" network has 0 events

Rafael de Souza Sanches · Answer 13 · Thu May 16 2024 04:04:26 GMT+0800 (China Standard Time)

Hey team, I've opened this other issue #886 because the error `insertLogFilterInterval" in this issue was already solved by the blockchain team, so I think this issue could be closed. We are still having problems to fetch the events, but it's related to other problems now. I appreciate your help with this issues!