Implement peers / network
yoshuawuyts opened this issue · comments
hypercore
now works pretty well, it's time to implement the networking part.
Todos
- implement
feed.announce()
(ref) - implement
feed.unannounce()
(ref) - implement
pub feed.has()
(ref) - implement
pub feed.has_all()
(polymorphic variant offeed.has()
) - implement
pub feed.head()
(ref) - implement
pub feed.replicate()
(ref) - implement
pub feed.update()
(ref) - implement
pub feed.downloaded()
(ref) - implement
pub feed.finalize()
(ref) - implement
Peer
struct - implement futures channel for events
- implement
Event
enum
Bonus
Skipping
Resources
Networking Vaporware API
let feed = Feed::default();
feed.append(b"data")?;
for event in feed {
match event {
Event::Data(data) => {},
Event::PeerAdd => {},
Event::PeerRemove => {},
Event::Download => {},
Event::Synchronize => {},
Event::Append => {},
}
}
What are you planning on using for networking?
std::net
tokio
- Are we waiting for official
async
/await
support? - ...
@pierd great question! -- I've bee thinking about this a bunch, even to the point where I've joined the networking WG to make more sense of the landscape.
I think there's no evading that we need tokio
+ async/await
support for this to work well. Luckily most of this should be in a usable state sometime soon, which is great! But there's no guarantees.
So I'm thinking for next steps it'd be good to start implementing the Dat networking layer. This would include extracting / re-building (if it isn't in libp2p
already) of the Kademlia Peer Selection structure https://github.com/ethereum/wiki/wiki/Kademlia-Peer-Selection, and the work our way up to implement the Dat networking layer. WIP references are available here dat-ecosystem-archive/DEPs#8 and here dat-ecosystem-archive/DEPs#7. Providing feedback to those would be super helpful too!
So yeah, there's a few steps to take here. I'd happily provide write access to datrs
if anyone wants to start building out parts of the networking layer (so we can keep the stack centralized, which is great for maintenance!).
Hope this all makes sense! -- Sorry for not communicating much of this earlier; only got a good grasp of the next steps to take here when I met up with mafintosh in Vancouver earlier this month.
References
I think that a nice addition would be to follow https://github.com/bluejekyll/trust-dns and their technical decisions when it comes to networking (https://github.com/bluejekyll/trust-dns/issues/569#issuecomment-433561439). IIRC dns multicast is also part of the swarm-discovery process (see https://www.npmjs.com/package/@hyperswarm/discovery)
Also worth mentioning: libp2p/libp2p#28 libp2p/rust-libp2p#590
Regarding networking, it seems that Dat is going to be migrating to hyperswarm once that's up and running.
I would like to help with this. I've started a small test to discover how mdns and libp2p works, and if it would interop with current dat on node, specially with the hyperswarm migration.
Any pointers to start/continue with this?
@bltavares I haven't done any research on the network protocol yet, so it probably requires scouring through the JS reference impl to find out. Doing that and sharing the notes would probably be the first step in making progress on this.
I'm thinking that a first version of the network protocol should probably tackle TCP first, and a single discovery mechanism. It can then later be expanded to cover more protocols once we have an MVP.
I've started a research project on the network interop on bltavares/colmeia, specially to understand how viable is to use existing rust crates for discovery.
My plan is to find a running dat repo on my local network, then find it use hyperswarm dht, and provide a list of ip:ports to connect for a given dat public key. (No url DNS lookup yet).
I will take this slowly, as much of my computer time is being used at work and I'm trying to take things slowly :)
I'll push my notes and discoveries on that repo :)
I played around with hypercore-protocol in Rust too and made it up to receiving a first data message from a regular nodejs hypercore replication stream, but with the NOISE handshaking and encryption disabled. See hypercore-protocol-rust-experiments and this PR to disable NOISE in hypercore.
I've finished this weekend with a barely functional dat server. It is broadcasting MDNS requests on the network, and it can be reached by npx dat share
on the same computer.
The handshake part is working, as well as the encryption. It needs now to implement the remaining protocol to exchange data.
I'll park this a bit, and try to get it cross-compiled to run on Android just to check if it is working :)
On another news: colmeia
runs in Android already! pics
I've started playing around with the idea of colmeia-clone
.
The design I'm coming from was the idea of having a Observer
structure that implements the network stack, and contains two hypercore
feeds. It did not work as hypercore
is not Send
.
The networks stack on hypercore API seems to live inside hypercore, given the .download
method, .replicate
, Peer
struct and the vaporware API example on events,
I can think of a couple of ideas:
- Implement the network stack trait on
hypercore
directly in attempt to match the JS implementation - Make
hypercore
send so it can be used on async-io fashion (might be needed to implement streams for the feed event api)
I'll try to write an async test case on hypercore to drive the changes to make it Send
. Is this feasible and a good action plan?
Any updates? I'm thinking of picking this up if nobody is working on it at the moment.