API not returning correct values for solidity uint256

Question

API not returning correct values for solidity uint256

halljson opened this issue a year ago · comments

I am trying to sync univ2 swaps and read them back out via the API.

The following query works (the results match what is on etherscan):

select * from swap 
where "tx_hash"='0x8cb77f390961b967d0fbbb4745a1f4aba6e86787372c95acdfafa062c04ee940' 
and amount1out=106531830891460120;

to: 0xf164fC0Ec4E93095b804a4795bBe1e041497b92a	
tx_hash: 0x8cb77f390961b967d0fbbb4745a1f4aba6e86787372c95acdfafa062c04ee940	
block_number: 10207859	
block_hash: 0x51876f00dab692e4229e42494c82ae53517b4edcb1bb2e27bea4ef534cf4ff8c	
timestamp: 1591388247
amount0in: 140
amount1in: 0
amount0out: 0
amount1out: 106531830891460120 // <-- wei equivalent of the "0.10653183089146012" eth value on etherscan

However, when I try to make the same query through the API, the amount1out is incorrect:

curl "localhost:3030/api/swap?limit=100&tx_hash=0x8cb77f390961b967d0fbbb4745a1f4aba6e86787372c95acdfafa062c04ee940&amount1out=106531830891460120" | jq

{
  "events": [
    {
      "amount0in": 140,
      "amount0out": 0,
      "amount1in": 0,
      "amount1out": 1838348756084981800,  // <-- incorrect
      "block_hash": "0x51876f00dab692e4229e42494c82ae53517b4edcb1bb2e27bea4ef534cf4ff8c",
      "block_number": 10207859,
      "contract_address": "0xa6f3ef841d371a82ca757fad08efc0dee2f1f5e2                        ",
      "indexed_id": "02704168-9a50-45aa-b465-fef8e103e9b1",
      "sender": "0xf164fC0Ec4E93095b804a4795bBe1e041497b92a",
      "timestamp": "1591388247",
      "to": "0xf164fC0Ec4E93095b804a4795bBe1e041497b92a",
      "tx_hash": "0x8cb77f390961b967d0fbbb4745a1f4aba6e86787372c95acdfafa062c04ee940"
    }
  ],
  "pagingInfo": {
    "next": "127.0.0.1:3030/swap?offset=100&limit=100&amount1out=106531830891460120&tx_hash=0x8cb77f390961b967d0fbbb4745a1f4aba6e86787372c95acdfafa062c04ee940",
    "previous": null
  }
}

I believe it has something to do with storing the values as NUMERIC data type, but I tried to tweak the value types to BIGINT to no avail.

It's possible that uint256 values may need to be stored as VARCHAR, or perhaps there's a better way.

Josh Stevens · Answer 1 · Thu Jul 13 2023 01:21:33 GMT+0800 (China Standard Time)

Hey can I have your config mapping and il fix it for us 👍

Josh Stevens · Answer 2 · Thu Jul 13 2023 01:23:54 GMT+0800 (China Standard Time)

Oh wait on so SQL is ok but API result is not ok? Ah yes I’m guessing that’s the case if I have your config I can sync a few and try myself - thanks for raising this it allows us to fix things quickly. Glad the events all matched up so let’s fix this API, I was planning on fixing the number issues and also allow you to do ranges between numbers etc

Jason Hall · Answer 3 · Thu Jul 13 2023 01:27:45 GMT+0800 (China Standard Time)

@joshstevens19

{
  ...,
  "fromBlockNumber": 10207858,
  "eventMappings": [
    {
      "syncBackRoughlyEveryNLogs": 1000,
      "decodeAbiItems": [
        {
          "anonymous": false,
          "inputs": [
            {
              "indexed": true,
              "internalType": "address",
              "name": "sender",
              "type": "address"
            },
            {
              "indexed": false,
              "internalType": "uint256",
              "name": "amount0In",
              "type": "uint256"
            },
            {
              "indexed": false,
              "internalType": "uint256",
              "name": "amount1In",
              "type": "uint256"
            },
            {
              "indexed": false,
              "internalType": "uint256",
              "name": "amount0Out",
              "type": "uint256"
            },
            {
              "indexed": false,
              "internalType": "uint256",
              "name": "amount1Out",
              "type": "uint256"
            },
            {
              "indexed": true,
              "internalType": "address",
              "name": "to",
              "type": "address"
            }
          ],
          "name": "Swap",
          "type": "event"
        }
      ]
    }
  ]
}

Josh Stevens · Answer 4 · Thu Jul 13 2023 01:32:32 GMT+0800 (China Standard Time)

Il look first thing for you and get a fix out

Josh Stevens · Answer 5 · Thu Jul 13 2023 15:39:37 GMT+0800 (China Standard Time)

The fix is above but bare in mind for fast queries you may need to add some custom indexes for example

CREATE INDEX IF NOT EXISTS swap_amount1out_idx__to ON swap("amount1out");

now makes that query instant without the index its 27 seconds. I am going to work on a way you can define custom indexers in the config for us to make this apply on creation

Josh Stevens · Answer 6 · Thu Jul 13 2023 15:39:51 GMT+0800 (China Standard Time)

note now big numeric numbers will return as strings in the responses

Josh Stevens · Answer 7 · Thu Jul 13 2023 16:27:56 GMT+0800 (China Standard Time)

#30

now you can apply custom indexes

Josh Stevens · Answer 8 · Thu Jul 13 2023 16:28:58 GMT+0800 (China Standard Time)

a few things to read:

https://github.com/joshstevens19/reth-indexer#applyindexesbeforesync---optional---default-false
https://github.com/joshstevens19/reth-indexer#setting-custom-db-indexes---customdbindexes---optional

let me know if you have any issues now these new features are supported

Jason Hall · Answer 9 · Fri Jul 14 2023 00:14:27 GMT+0800 (China Standard Time)

It's working for me, thank you for this.

BTW - it's about 5 hours to sync about 75GB of univ2 swaps, new pairs, and LP mint events - impressive stuff.

Really like the custom indexes too: timestamp was the first thing I added.

Josh Stevens · Answer 10 · Fri Jul 14 2023 00:31:36 GMT+0800 (China Standard Time)

Awesome!! I think we can make it even faster soon that’s great to hear it synced that much data 🦾🦾