Optimize sync and make it real-time
frankhinek opened this issue · comments
Background/Context
During a sync pull()
operation:
- a
MessagesGet
request is sent over the network to the remote DWN for everymessageCid
that needs to be sync'd - if the message returned by
MessagesGet
does NOT containencodedData
, then an additionalRecordsRead
must be sent over the network to fetch the data from the datastore, if any
During a sync push()
operation:
- a
MessagesGet
request is sent to the local DWN for everymessageCid
that needs to be sync'd - if the message returned by
MessagesGet
does NOT containencodedData
, then an additionalRecordsRead
must be processed by the local DWN to fetch the data from the datastore, if any
An obvious optimization we can and will pursue is to batch the first step such that all messageCids
are included in a single MessagesGet
request sent either to the local DWN or over the network to a remote DWN. This is a change to an application or SDK using dwn-sdk-js
, so as such, will be implemented first in the web5-js
SDK.
However, as currently implemented, there isn't a way to optimize the second step.
Question to Research and Propose Alternative Solutions
How might we modify MessagesGet
(or add a new method) such that fewer over the network requests are needed to retrieve the message metadata and data payload for each message that is queued for sync?
Can we just use/require HTTP/2 framed multiplexing for multiple file fetch?