Electrum: supports batching?

Question

Electrum: supports batching?

tiero opened this issue 2 years ago · comments

I am trying to send a batch of requests to receive a batched response, using wss://blockstream.info/liquidtestnet/electrum-websocket/api endpoints, but as soon I send an array instead of an object over the socket it disconnects.

It works!

{"jsonrcp":"2.0","id":1,"method":"blockchain.scripthash.get_history","params":["c40a8fadd3f13a91d506b820b1158cbb8166a2f45345088843d8f8483d44e17e"]}

Does not works

[
  {"jsonrcp":"2.0","id":12103,"method":"blockchain.scripthash.get_history","params":["c40a8fadd3f13a91d506b820b1158cbb8166a2f45345088843d8f8483d44e17e"]},     
  {"jsonrcp":"2.0","id":12104,"method":"blockchain.scripthash.get_history","params":["57f6e1e3b30eea697504172e2a7967e3cb1f0dac5a6a188d18fa14d5fde1b850"]}
]

Marco Argentieri · Answer 1 · Wed Dec 14 2022 23:23:23 GMT+0800 (China Standard Time)

Apparently also using normal JSONRPC over TCP I can't use batching, the connection closes regardless when sending an array.

It works!

echo '{"jsonrpc": "2.0", "method": "blockchain.scripthash.get_history", "id": 0, "params":["c40a8fadd3f13a91d506b820b1158cbb8166a2f45345088843d8f8483d44e17e"]}' | ncat --ssl blockstream.info 465

Does not work

echo '[{"jsonrpc": "2.0", "method": "blockchain.scripthash.get_history", "id": 0, "params":["c40a8fadd3f13a91d506b820b1158cbb8166a2f45345088843d8f8483d44e17e"]}, {"jsonrpc": "2.0", "method": "blockchain.scripthash.get_history", "id": 0, "params":["57f6e1e3b30eea697504172e2a7967e3cb1f0dac5a6a188d18fa14d5fde1b850"]}]' | ncat -v --ssl blockstream.info 465
Ncat: Version 7.93 ( https://nmap.org/ncat )
Ncat: SSL connection to 35.201.74.156:465.
Ncat: SHA-1 fingerprint: A702 951A 77CF 6FCD 5667 C047 EE43 82AA CAB0 7926
Ncat: 311 bytes sent, 0 bytes received in 0.19 seconds.

You can also replicate it with Node.js for example

https://gist.github.com/tiero/5b62ab65dab75a8b8367149e73c291ca

Nadav Ivgi · Answer 2 · Sat Dec 17 2022 12:53:39 GMT+0800 (China Standard Time)

Hi tiero! Thanks for reporting this.

The Electrum RPC doesn't currently support jsonrpc batching, it wasn't supported in romanz/electrs when blockstream/electrs forked off of it and we never added it.

It might be worth mentioning that its not clear (to me at least) that batching is actually better than sending multiple separate jsonrpc requests over a long-lived TCP connection (in fact its theoretically worse with batching because the server has to buffer all the responses). Electrum itself doesn't use batching.

That being said, supporting it should be relatively straightforward, and it does make sense at least for compatibility even if there are no other direct gains. I'll look into this.

Nadav Ivgi · Answer 3 · Sat Dec 17 2022 12:57:54 GMT+0800 (China Standard Time)

To expand on my previous comment, what I mean by 'send multiple separate requests' is that instead of:

[ { "id":1,"method":"foo",... }, { "id":2,"method":"bar",... } ]

which is what you would do with batching, you can just send multiple separate JSON objects separated with newlines and no wrapping array, like this:

{ "id":1,"method":"foo",... }
{ "id":2,"method":"bar",... }

You will then similarly get the responses back as newline-delimited JSON objects instead of an array.

The overall effect is very similar, at least when used over a long-lived TCP connection (if the client opens a separate connection for each request, batching would improve things). But it has the added benefit that the server can process the RPC requests in parallel and send each response when its ready [0], without having to buffer all the responses first to turn them into an array. This should improve memory consumption on the server's side and result in faster response times (they get streamed live instead of returned all at once after everything is processed).

[0] The current implementation doesn't actually process RPC requests from the same client in parallel, but it theoretically could. It does, however, return the responses in serial as they're available and doesn't buffer them.

Marco Argentieri · Answer 4 · Mon Dec 19 2022 04:31:15 GMT+0800 (China Standard Time)

Thanks!

Would this work with WebSocket interface too? Is this correct way eventually?

<html>

<head>
  <title>My App</title>
</head>

<body>
  <script type="module">
    const ws = new WebSocket("wss://blockstream.info/liquidtestnet/electrum-websocket/api");
    ws.onopen = () => {
      const requests = [
        { method: 'blockchain.scripthash.get_history', params: ['c40a8fadd3f13a91d506b820b1158cbb8166a2f45345088843d8f8483d44e17e'] },
        { method: 'blockchain.scripthash.get_history', params: ['57f6e1e3b30eea697504172e2a7967e3cb1f0dac5a6a188d18fa14d5fde1b850'] },
        { method: 'blockchain.scripthash.get_history', params: ['231e179049214223eaa5a42c228bb50066d8f7d7711e47d2e05db2cfc600d584'] }
      ];
      let id = Math.ceil(Math.random() * 1e5);
      let payload = requests.map(({ method, params }) => {
        id++;
        return prepareRequest(id, method, params);
      }).join('\r\n');

      console.debug('Electrum WS request: ', payload);
      ws.send(payload);
    };

    ws.onerror = console.error;
    ws.onclose = console.error;
    ws.onmessage = (event) => {
      const { result, error } = JSON.parse(event.data);
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    };

    function prepareRequest(id, method, params) {
      return JSON.stringify({ jsonrpc: '2.0', id, method, params });
    }

  </script>
</body>

</html>

Nadav Ivgi · Answer 5 · Mon Dec 19 2022 12:49:35 GMT+0800 (China Standard Time)

This is mostly correct, however:

Note that the URL should be wss://blockstream.info/liquidtestnet/electrum-websocket/ (no /api)
For some reason I couldn't get it to work with a single ws.send() for all the requests (I believed it should work, not sure why it doesn't). But I was able to make it work using multiple send() calls, with requests.forEach(({ method, params }) => ws.send(prepareRequest(++id, method, params)+'\n'))
I'm assuming the resolve()/reject() calls are for a Promise? If so note that promises can only be resolved with a single value, so this wouldn't quite work

https://jsfiddle.net/n3c12f7m/1/

Marco Argentieri · Answer 6 · Mon Dec 19 2022 18:55:58 GMT+0800 (China Standard Time)

For some reason I couldn't get it to work with a single ws.send()

Thank you! indeed sending in a single ws.send was failing to me. I assume since send are done in a single ws.onopen it should not close the channel on each requests.

If so note that promises can only be resolved with a single value, so this wouldn't quite work

Good point. Maybe the best would be to keep a counter of req/resp and only resolve the promise when the entire batch of requests has results and return an array.

Marco Argentieri · Answer 7 · Mon Dec 19 2022 19:31:57 GMT+0800 (China Standard Time)

@shesek It will work even if you do not put the newline \n

This works just fine.

requests.forEach(({ method, params }) => ws.send(prepareRequest(++id, method, params)))

I guess is websocat or jsonrpc client is adding the \n by default?

Nadav Ivgi · Answer 8 · Thu Dec 22 2022 07:36:55 GMT+0800 (China Standard Time)

Good point. Maybe the best would be to keep a counter of req/resp and only resolve the promise when the entire batch of requests has results and return an array.

An alternative is to return an event emitter that you can subscribe on to be notified when new responses arrive, or accept a callback that gets called with each response. This only makes sense if your app has something useful to do with partial responses though (but I guess that at the very least it could be useful in order to display a progress indicator to the user)

I guess is websocat or jsonrpc client is adding the \n by default?

Interesting. Judging by the Rust code it seems like it shouldn't work without it, so it might indeed be websocat that adds it. One way to tell for sure is to see if this works when communicating with the (non-WS) Electrum RPC directly