Polish nodejs bindings

[George Kadianakis]

We merged the nodejs bindings in #147 but there is still polishing work to be done.
Please look at #147 for unresolved issues.

Most importantly:

• the VerifyBlobKzgProofBatch() function is quite messy. How can we simplify it? (See #147 for some ideas)
• verification checks are split between the C code and the typescript code. How can we unify them?

/cc @dapplion @dgcoffman

[Lion - dapplion]

CC @matthewkeil to check if you have avail to look into these pending items

[Matthew Keil]

My pleasure @dapplion. Hello @asn-d6! My name is Matt and I would love to help out with this issue. I reviewed the code and PR #147. I have a couple questions to ask, some code samples to show, and some API options you may select from to dial in the final product.

In regards to the first two questions above I think your intuition is correct.

• The manually memory management and memcpy in VerifyBlobKzgProofBatch may not be the best approach while on the node thread. There are ways to obtain the native pointer from the byte array that comes in from JS, and only hold a reference to them while the native library is using the data.
• For the verification checks, are you referring to prop validation happening across contexts like in this comment?. If so, yes there is a way to do that natively so that code context is preserved for maintainers. It will also, arguably, run marginally faster when done natively.

A couple of notes about #147 and the API

• @asn-d6 you are correct in this comment. It is possible. Its also probably a very good idea, to refactor napi_typed_array_from_bytes to avoid the uint8_t* -> std::vector<uint_8t> -> Napi::ArrayBuffer -> Napi::Uint8Array from both the error/memory safety perspective and for performance reasons.
• this may be an issue and will segfault if the setupHandle gets overwritten somehow. It's instantiated with let. It possible to store the KZGSettings as a native pointer, SetInstanceData, so it doesn't have to be passed with execution calls to guard against this.
• this issue got fixed but with a quick glance there is a potential double free on #L338 that has a goto out; and another free on #L387

As for potential opportunities I have some code samples to show you. All of the samples below were taken from @chainsafe/blst-ts. Its a similar type of project. I am still debugging a few things in there but it will be helpful as an example. I also have some other bindings libraries I can point you to if the code style is not to your liking (ie its c++ post 11 heavy with little natural c like macros/malloc/etc)

• this is the index.js entrance file
• this is the index.d.ts typings file
• this is a class that helps with arg parsing. The constructor implementation can be found here and an example of the usage of the class can be seen here and here

Turning to API choices, I noticed that these functions are designed to run synchronously on the main thread. This type of heavy computation is generally better to do off of the main JS thread. Is a Promise based API something you would like to have implemented? Its possible to do only the argument parsing and value return (that require the engine) while on the main thread but to do the heavy math on a worker thread. Here is a class that implements the Napi::AsyncWorker and an example of its use for a similar type of problem. What are your thoughts?

[George Kadianakis]

My pleasure @dapplion. Hello @asn-d6! My name is Matt and I would love to help out with this issue. I reviewed the code and PR #147. I have a couple questions to ask, some code samples to show, and some API options you may select from to dial in the final product.

Hey Matt! Glad to have you on board! Thanks a lot for the detailed comment! Do keep in mind that I don't know much about node or js so please take all my comments with a grain of salt :-)

In regards to the first two questions above I think your intuition is correct.

* For the verification checks, are you referring to prop validation happening across contexts like in [this comment?](https://github.com/ethereum/c-kzg-4844/pull/147#discussion_r1114368382).  If so, yes there is a way to do that natively so that code context is preserved for maintainers.  It will also, arguably, run marginally faster when done natively.

Great! I don't have a strong opinion on whether these checks should happen in javascript or C, but I would like to be consistent and intentional on where we do these checks, so that we don't miss any of them.

So if we decide to split them up (as they are right now) we should think of how to to make this clear to the reader (because I totally missed it). With regards to speed, I imagine that these checks are very quick to do both in JS and in C, so I'm not too worried about performance.

A couple of notes about #147 and the API

* @asn-d6 you are correct in [this comment](https://github.com/ethereum/c-kzg-4844/pull/147#discussion_r1114182798). It is possible.  Its also probably a very good idea, to refactor `napi_typed_array_from_bytes` to avoid the  `uint8_t* -> std::vector<uint_8t> -> Napi::ArrayBuffer -> Napi::Uint8Array` from both the error/memory safety perspective and for performance reasons.

Great! Let's go with that if possible!

* [this may be an issue](https://github.com/ethereum/c-kzg-4844/pull/147#discussion_r1114171554) and will segfault if the `setupHandle` gets overwritten somehow.  It's instantiated with `let`.  It possible to store the `KZGSettings` as a native pointer,  [SetInstanceData](https://github.com/nodejs/node-addon-api/blob/main/doc/env.md#setinstancedata-1), so it doesn't have to be passed with execution calls to guard against this.

That's great to hear as well!

* [this issue got fixed](https://github.com/ethereum/c-kzg-4844/pull/147#discussion_r1114154791) but with a quick glance there is a potential double free on [#L338](https://github.com/ethereum/c-kzg-4844/blob/6b2ee20102abccd2e1f49cbe7be18d2a59698793/bindings/node.js/kzg.cxx#L338) that has a `goto out;` and another free on [#L387](https://github.com/ethereum/c-kzg-4844/blob/6b2ee20102abccd2e1f49cbe7be18d2a59698793/bindings/node.js/kzg.cxx#L387)

Ugh, good catch! I imagine that with the refactor of that function we won't need those frees anymore, but if we still need them, let's make sure we clean that bug :)

As for potential opportunities I have some code samples to show you. All of the samples below were taken from @chainsafe/blst-ts. Its a similar type of project. I am still debugging a few things in there but it will be helpful as an example. I also have some other bindings libraries I can point you to if the code style is not to your liking (ie its c++ post 11 heavy with little natural c like macros/malloc/etc)

* this is the [index.js](https://github.com/matthewkeil/blst-ts/blob/rebuild-bindings/lib/index.js) entrance file

* this is the [index.d.ts](https://github.com/matthewkeil/blst-ts/blob/rebuild-bindings/lib/index.d.ts) typings file

* [this is a class](https://github.com/matthewkeil/blst-ts/blob/0ea4f26aad0233d8fb29178e774d6d00a40a0762/src/addon.h#L60) that helps with arg parsing.  The constructor implementation can be found [here](https://github.com/matthewkeil/blst-ts/blob/0ea4f26aad0233d8fb29178e774d6d00a40a0762/src/addon.cc#L62) and an example of the usage of the class can be seen [here](https://github.com/matthewkeil/blst-ts/blob/0ea4f26aad0233d8fb29178e774d6d00a40a0762/src/functions.cc#L128) and [here](https://github.com/matthewkeil/blst-ts/blob/0ea4f26aad0233d8fb29178e774d6d00a40a0762/src/functions.cc#L157)

OK so I don't really have a strong opinion here (also those links are C++ code which I don't really speak). Please take the liberty to implement this in any way you prefer, as long as it's maintainable and easy to read (for the auditor).

Turning to API choices, I noticed that these functions are designed to run synchronously on the main thread. This type of heavy computation is generally better to do off of the main JS thread. Is a Promise based API something you would like to have implemented? Its possible to do only the argument parsing and value return (that require the engine) while on the main thread but to do the heavy math on a worker thread. Here is a class that implements the Napi::AsyncWorker and an example of its use for a similar type of problem. What are your thoughts?

No strong opinion here either I'd say. I'm not sure how much mental overhead a Promise-based API will add to the current implementation.

Perhaps one way to approach this, is to clean up the current node binding codebase and make it nice and correct. And after we do that, we can see how complex it is and whether it's worth going to a promise-based approach.

Here I'm assuming that "refactor and tidy up codebase" and "make it promise-based" are too mostly orthogonal tasks.

---

BTW, @matthewkeil here are two additional tickets that might need some love #149 and #118 .

Let me know if my responses above are not robust enough, or if you need more help. We also have a telegram group you might want to join. My telegram handle is asn_d6.

Cheers!

[Justin Traglia]

Excellent feedback @matthewkeil, thanks! I don't have much to add, maybe just my opinion on this:

Great! I don't have a strong opinion on whether these checks should happen in javascript or C, but I would like to be consistent and intentional on where we do these checks, so that we don't miss any of them.

If possible, I think these checks should happen in C. I spent a little time trying to get it working but couldn't figure it out. Someone with Node.js experience (like yourself) could do this properly.

[Lion - dapplion]

I agree the checks should happen in C land and keep the JS code to the bare minimum required for correctness. Going for promise based is a good path but we should explore that latter. We need data on how that influences with the usage patterns that are changing (See "Free the blobs" initiative). So let's tiddy first, async second

[Matthew Keil]

@asn-d6 I reworked the arguments in #170 but was unable to remove the memcpy. The array elements in JS will never be contiguous in memory (always heap allocated buffers passed by reference/pointer) and verify_blob_kzg_proof_batch is build for an array of Blobs. Would it be possible to refactor that function to take arrays of pointers and not arrays of structs? I did a bit of research and it looks like a few of the bindings can be easily switched but not sure that will be the case with all. What are your thoughts?

[George Kadianakis]

@asn-d6 I reworked the arguments in #170 but was unable to remove the memcpy. The array elements in JS will never be contiguous in memory (always heap allocated buffers passed by reference/pointer) and verify_blob_kzg_proof_batch is build for an array of Blobs. Would it be possible to refactor that function to take arrays of pointers and not arrays of structs? I did a bit of research and it looks like a few of the bindings can be easily switched but not sure that will be the case with all. What are your thoughts?

Hmm, I see.

I don't think we should change the interface of verify_blob_kzg_proof_batch() from const Blob *blobs to const Blob **blobs as this would be pretty non-standard, and also complicate the internal KZG library functionality.

It's sad that contiguous memory arrays are not supported in nodejs as they are supported in other langs like Rust which allows the bindings to be much simpler.

The other approach would be to have the nodejs client code flatten the list of blobs into a single flattened super-blob and pass that to the binding as a uint8 array, but that's also not intuitive or nice.

[Matthew Keil]

I don't think we should change the interface of verify_blob_kzg_proof_batch() from const Blob *blobs to const Blob **blobs as this would be pretty non-standard, and also complicate the internal KZG library functionality.

Definitely makes sense. Thanks for the consideration!!

The other approach would be to have the nodejs client code flatten the list of blobs into a single flattened super-blob and pass that to the binding as a uint8 array, but that's also not intuitive or nice.

I also thought about doing the flattening at the js layer before passing in (like the verification checks were) so that the API was still what a consumer would "expect," but no matter how its sliced there is a copy somewhere. I figured at least this way its methodical and not "node magic" allocation that will need to wait for a GC cycle to actually free. I added the HandleScope to the loop context in the off chance that a GC cycle happens mid run it will remove the temporary values. But that was more of a best practice thing than something that feels like it will make a meaningful difference.

[George Kadianakis]

Thanks a lot @matthewkeil !!!! Closing this one.