Bulk lookup proofs
slawlor opened this issue · comments
In production, under high-load we might need the ability to perform "bulk" lookup proofs. This means that we should do the same breadth-first-search for a collection of users as is done under the Directory::publish operation (as is done in append_only_zks.rs:191).
This means, when doing a lookup we should calculate all the prefixes and pre-load (via BFS walk of batch-retrievals) all the nodes along the path into the cache. Then we can generate the proofs without doing many individual retrievals.
Looking at the preload_nodes_for_insertion function, it retrieves the nodes from storage but I am not sure how it achieves the caching we would like to have. I might be missing something here but the function does not seem to keep the nodes, e.g., in memory. How do we want to cache the nodes for bulk lookup proofs?
Hey @eozturk1, so in the MySQL storage layer we already have a caching scheme implemented. By using the preload_nodes_for_insertion you're effectively using batch get operations rather than individual get ops. Anytime a record is retrieved, we're appending it to the cache so you don't need to worry about manually inserting them into any cache, it's auto-magic ;)
The bulk-get is MUCH fewer db ops which is why it's helpful to retrieve them all before, then after that it's just memory operations which are super cheap. Even though we're double-getting data, it's still way more performant and makes the code transparent to the AKD dev.
Ah, that's perfect! Just saw the TimedCache.:) I'll use the same batch ops for bulk proofs.