add method to rfc6962 hasher to read from io.Reader
bobcallaway opened this issue · comments
Add convenience method to RFC6962 hasher to accept io.Reader.
This allows a hash to be computed for a stream (e.g a large file) without holding the entire object in memory.
How would that be used?
In Trillian, there are definitely size limitations, and in particular, the content to be hashed comes via an RPC, so is unavoidably all in memory, and large objects are almost certainly going to cause significant problems.
Is the goal to fix those limitations?
There are two APIs (GetInclusionProofByHash, GetLeavesByHash) where a Trillian RPC client needs to send the hash value to the server (not the actual content). The proposed convenience method is a client optimization for when the client would prefer to compute the hash value to send over RPC without storing the object to be hashed into memory.
How big is your data? Trillian wasn't designed to store large files, it's better if your data is O(kilobytes) at most. If your data is bigger than that you might need to consider storing some digest / identifier instead.
Surely you can compute the hash locally for the "read by hash" requests, but you still need this data written to the storage first. And for that you send a write RPC which results in your data (large file) being processed in memory.
@pavelkalinnikov fair point - thanks!