Giters

jungomi / xxhash-wasm

A WebAssembly implementation of xxHash

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Performance comparison

swernerx opened this issue · comments

commented

Hi! I made some performance evaluation of different hashing solutions for my library asset-hash. The results look pretty good for this library, and I thought about sharing it with you. Thanks a lot for the great work. Indeed, XXHash-WASM will be the default "backend" in the next major version.

commented

Hello, that is great to hear. For your information, some big improvements just landed from the PR #26, which will be in v1.0.0 that will be published in the next few days. That brings about a 3-4x performance improvement (especially for the 64-bit hash) and also a streaming API, inline with the crypto streaming API. From the quick look of the README of asset-hash, it looks like the streaming API is something you also incorporate.
The only caveat, is that it uses some newer features, which require fairly recent versions of the browsers/Node, as mentioned in https://github.com/jungomi/xxhash-wasm#engine-requirements:

Engine Requirements

In an effort to make this library as performant as possible it makes use of several recent additions to browsers, Node, and the Webassembly specification. Notably, this includes:

  1. BigInt support in WebAssembly
  2. Bulk memory operations in WebAssembly
  3. TextEncoder.encodeInto

Taking all of these requirements into account, xxhash-wasm should be compatible with:

  • Chrome >= 85
  • Firefox >= 79
  • Safari >= 15.0
  • Node >= 15.0

If support for an older engine is required, xxhash-wasm 0.4.2 is available with more limited engine requirements, with 3-4x slower hashing performance.

commented

Thanks for the immediate detailed response. Indeed this additional performance improvement would be very nice! And yes, streaming is also a good choice for hashing larger assets.

If you don't mind... would it be possible to implement some built-in base-encoding into the library (based on WASM?) - that way we could offer Base58 / Base52 (usable for file names) without libraries like BigInt etc. ... should be another performance improvement overall.

commented

Node v16 would be fine as a runtime requirement for the next major version!

commented

If you don't mind... would it be possible to implement some built-in base-encoding into the library (based on WASM?) - that way we could offer Base58 / Base52 (usable for file names) without libraries like BigInt etc. ... should be another performance improvement overall.

The encoding is out of scope for this library, this is purely for the hashing of a given sequence of bytes. The only "encoding" uses TextEncoder, which is not the same kind of encoding as you are referring to, because it's essentially just getting the memory representation of strings.
If you want to generate the bytes differently without having to create a string from it, you can do that and then use the h32Raw/h64Raw methods, which accept a Uint8Array (and the streaming API also accepts Uint8Arrays).

I think it would also generally make more sense to have a specialised/optimised library for such encodings, which could be implemented in WASM to get better performance, and you could use the resulting bytes as Uint8Array directly without having to convert it to a string. With that, there won't really be any additional overhead except for copying it to the corresponding memory, which would be the only potential benefit I can think of, to have the encoding included in this library.

commented

Thanks!

I wonder if there is already a plan to ship v1.0. Indeed I would need this release for continue working on the new asset-hash version.

commented

v1.0.0 has just been published. You can see the changes in https://github.com/jungomi/xxhash-wasm/releases/tag/v1.0.0

commented

I guess I close this for now. Thanks a lot!