Remote interface: Design asynchronous N-N remote PoW
ajblane opened this issue · comments
+--------------------------------------------------+
| Client |
| +--------------------+ +----------------------+ |
| |asynchronous request| | wait for result event| |
| +--------------------+ +----------------------+ |
| _______________________________ |
| |correlation id |trytes| status| |
| |---------------|------|-------| |
| |_______________|______|_______| |
| |
+-----------|------------------------^-------------+
+-----------|------------------------|-------------+
| v RabbitMQ broker | |
| +------------------+ +------------------+ |
| | incoming queue | | completed queue | |
| | | | | |
| | - trytes | | - PoW result | |
| | - mwm | | | |
| | | | | |
| +------------------+ +------------------+ |
| | ^ |
+-----------|----------------------- |-------------+
v |
+--------------------------------------------------+
| asynchronous reply - dcurl multi-workers |
+--------------------------------------------------+
In asynchronous request and reply pattern, multi-workers do not orderly reply to client requests. We require a FIFO buffer to record correlation id, trytes and state per request to provide information for ensuring getting the PoW results to broadcast is ordering and also require a lock to manage the FIFO buffer to avoid operating it at the same time.
Here is a brief illustration of how it works. When sending asynchronous requests, we orderly record timestamp, correlation id and state in client. if three requests are processed in three workers and their works are not yet done, the FIFO buffer in client is as follows:
___________________________________
|timestamp| correlation id| status|
|---------|---------------|-------|
|____t1___|_______id1_____|_______|
|____t2___|_______id2_____|_______|
|____t3___|_______id3_____|_______|
If two workers rely PoW results to client and correlation ids separately are id1 and id3, the FIFO buffer is as follows:
___________________________________
|timestamp| correlation id| status|
|---------|---------------|-------|
|____t1___|_______id1_____|__done_|
|____t2___|_______id2_____|_______|
|____t3___|_______id3_____|__done_|
At this moment, we can broadcast the PoW result of id1 and not broadcast the PoW result of id3. After broadcasting the PoW result of id1, we need to send the ack of id1 to the broker and not need to send the ack of id3. This is because it can avoid losing the id3 message when unexpectedly shutting down client. The FIFO buffer is as follows:
___________________________________
|timestamp| correlation id| status|
|---------|---------------|-------|
|____t2___|_______id2_____|_______|
|____t3___|_______id3_____|__done_|
If the status of the PoW result of id2 is done, we can continuously broadcast the PoW results of id2 and id3 and send acks of id2 and id3 to the broker. The FIFO buffer may be as follows:
___________________________________
|timestamp|corrleration id| status|
|---------|---------------|-------|
|____t4___|_______id4_____|_______|
|____t5___|_______id5_____|_______|
By the way, if the PoW result of id4 is a poison message, we can use local dcurl to compute PoW for this request and remove the information of id4 while the FIFO buffer needs to be locked.
Client-side interface is expected as follows:
public:
bool remote_interface_init(char *ip, broadcast())
- connect to the RabbitMQ broker with ip address and default port
- initialize the FIFO buffer
- initialize the lock
- remote_interface_asyn_reply(broadcast())
void remote_interface_asyn_request(int8_t *trytes, int mwm)
- lock before operating the FIFO buffer
- push correlation id (UUID) and tryes of this request into the FIFO buffer
- unlock after operating the FIFO buffer
- set correlation id and reply-to properties for the request
- send the request to the RabbitMQ broker
void remote_interface_destroy()
- destroy the FIFO buffer
- leave the RabbitMQ broker
private:
void remote_interface_asyn_reply(broadcast())
- wait for the result event in for loop
- after receiving the result, we check whether correlation id can be broadcast
- if the reply can be broadcast, we check whether the reply is a poison message. If the reply is a poison message, we use dcurl. And if the reply is not a poison message, the reply is broadcast. Whatever we use dcurl or the reply is broadcast, they lock the FIFO, pop data form the FIFO and then unlock FIFO
- if correlation id of the reply is not matched, we search the position of correlation id in the FIFO and record status
In this article, "Using Distributed Computing at Scale", the author describes that with his architecture he was able to run the system under the workload on over 2 million tasks a day which can be scaled easily.
By the above article, I want to replace the completed queue into Redis. There are some reasons as follows:
- The type of RabbitMQ queue is FIFO and we can not control the order of receiving results, so in client, we require a self-managed FIFO queue to record order number of transactions and a modified behavior of popping the queue for order. For storing PoW results into Redis by workers, we can store results in order to Redis by the RabbitMQ-provided delivery tag
- In client, we only need to get results from Redis by holding order number
+--------------------------------------------------+
| Client |
| +--------------------+ +----------------------+ |
| |asynchronous request| | search Redis | |
| +--------------------+ | by order number | |
| +----------------------+ |
+-----------|------------------------^-------------+
+-----------|------------------------|-------------+
| v RabbitMQ broker | |
| +------------------+ +------------------+ |
| | incoming queue | | Redis | |
| | | | | |
| | - trytes | | 0 | PoW result | |
| | - mwm | | 1 | PoW result | |
| | | | 2 | PoW result | |
| | | | . | PoW result | |
| +------------------+ +------------------+ |
| | ^ |
+-----------|----------------------- |-------------+
v |
+--------------------------------------------------+
| asynchronous reply - dcurl multi-workers |
+--------------------------------------------------+
In addition to remote interface, you shall describe the inference how workers communicate to existing dcurl-based implementation context.
In addition to remote interface, you shall describe the inference how workers communicate to existing dcurl-based implementation context.
This above inference may be complex to be used in the scenario of AttachToTangle API. The issue, "Design remote PoW-accelerated IRI's AttachToTangle API #133", describes the features of AttachToTangle API we care about and the scenario of using remote interface. There is a RPC pattern provided by RabbitMQ to be used in the scenario of AttachToTangle API.
Therefore, I think the remote interface can be same as dcurl interface. Here is a roughly description.
- bool dcurl_init()
- connect to the RabbitMQ broker with ip address and default port
- int8_t *dcurl_entry(int8_t *trytes, int mwm, int threads)
- set correlation id and reply-to properties for the request
- consume the broker queue, check correlation id, and return PoW result of this request
- void dcurl_destroy()
- leave the RabbitMQ broker