Performance comparison
swernerx opened this issue · comments
Hi! I made some performance evaluation of different hashing solutions for my library asset-hash. The results look pretty good for this library, and I thought about sharing it with you. Thanks a lot for the great work. Indeed, XXHash-WASM will be the default "backend" in the next major version.
Hello, that is great to hear. For your information, some big improvements just landed from the PR #26, which will be in v1.0.0
that will be published in the next few days. That brings about a 3-4x performance improvement (especially for the 64-bit hash) and also a streaming API, inline with the crypto
streaming API. From the quick look of the README of asset-hash
, it looks like the streaming API is something you also incorporate.
The only caveat, is that it uses some newer features, which require fairly recent versions of the browsers/Node, as mentioned in https://github.com/jungomi/xxhash-wasm#engine-requirements:
Engine Requirements
In an effort to make this library as performant as possible it makes use of several recent additions to browsers, Node, and the Webassembly specification. Notably, this includes:
BigInt
support in WebAssembly- Bulk memory operations in WebAssembly
TextEncoder.encodeInto
Taking all of these requirements into account,
xxhash-wasm
should be compatible with:
- Chrome >= 85
- Firefox >= 79
- Safari >= 15.0
- Node >= 15.0
If support for an older engine is required,
xxhash-wasm
0.4.2 is available with more limited engine requirements, with 3-4x slower hashing performance.
Thanks for the immediate detailed response. Indeed this additional performance improvement would be very nice! And yes, streaming is also a good choice for hashing larger assets.
If you don't mind... would it be possible to implement some built-in base-encoding into the library (based on WASM?) - that way we could offer Base58 / Base52 (usable for file names) without libraries like BigInt etc. ... should be another performance improvement overall.
Node v16 would be fine as a runtime requirement for the next major version!
If you don't mind... would it be possible to implement some built-in base-encoding into the library (based on WASM?) - that way we could offer Base58 / Base52 (usable for file names) without libraries like BigInt etc. ... should be another performance improvement overall.
The encoding is out of scope for this library, this is purely for the hashing of a given sequence of bytes. The only "encoding" uses TextEncoder
, which is not the same kind of encoding as you are referring to, because it's essentially just getting the memory representation of strings.
If you want to generate the bytes differently without having to create a string from it, you can do that and then use the h32Raw
/h64Raw
methods, which accept a Uint8Array
(and the streaming API also accepts Uint8Array
s).
I think it would also generally make more sense to have a specialised/optimised library for such encodings, which could be implemented in WASM to get better performance, and you could use the resulting bytes as Uint8Array
directly without having to convert it to a string. With that, there won't really be any additional overhead except for copying it to the corresponding memory, which would be the only potential benefit I can think of, to have the encoding included in this library.
Thanks!
I wonder if there is already a plan to ship v1.0. Indeed I would need this release for continue working on the new asset-hash version.
v1.0.0
has just been published. You can see the changes in https://github.com/jungomi/xxhash-wasm/releases/tag/v1.0.0
I guess I close this for now. Thanks a lot!