ponder-sh / ponder

A backend framework for crypto apps

Home Page:https://ponder.sh

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add indices to speed up `deleteRealtimeData` store method

0xOlias opened this issue · comments

I've observed timeouts in the deleteRealtimeData query when running against a large database (~20M rows in logs). The queries in that database transaction use the following columns:

blocks.chainId
blocks.number (already have an index)

transactions.chainId
transactions.blockNumber

logs.chainId (already have an index)
logs.blockNumber

rpcRequestResults.chainId
rpcRequestResults.blockNumber

logFilterIntervals.startBlock
logFilterIntervals.endBlock

factoryLogFilterIntervals.startBlock
factoryLogFilterIntervals.endBlock

I have a hunch that as long as each table has an index on block number (only one for the interval tables), we'll be fine. The query always passes a block number very close to tip, so if the planner is able to use that index first, scans are probably fine for the remainder (eg chain ID).

Another (creative, heh) idea would be to take advantage of the existing blocks.number index.

DELETE FROM "transactions" where "blockHash" in (SELECT "hash" from "blocks" WHERE "number" > $1);

I think this would get us the same planner optimization described above without adding an index on transactions.blockNumber (which would only get used in this one infrequent query).

Before moving forward with a fix, it would be smart to isolate the slowest query within the transaction.