Callback when infohashes are seen
zond opened this issue · comments
I have recently been using Popcorn Time a lot, and I like the features but dislike the build quality.
So I started thinking about how one could build something similar, but more robust. The first improvement, of course, would be to build it in Go instead of JavaScript.
Another would be to use some other search/browse database than whatever Popcorn Time uses (which isn't very robust at all).
Finally I found https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf, which describes how to seed the network with pure forwarding nodes that only tap the infohashes it sees in those nodes, and then fetch the metadata from the hosting nodes to put them in a local searchable database.
This made me start looking at your code. Unfortunately I get confused by it - I am used to chord-style networks, and kademlia is like greek to me :)
Where should I add code to enable some kind of callback function whenever the node gets queried about an infohash?
Look at the Logger interface. When creating your dht server, set dht.Logger to the thing you want to operate.
See:
https://github.com/nictuku/dht/blob/master/dht.go#L218
and the doc:
http://godoc.org/github.com/nictuku/dht#Logger
Sorry that it's not documented. Feel free to send a patch ;-).
Holy shit, I randomly noticed your username. I'm Yves, we just met today at work. Small world ;-)
Ah, yeah. I actually managed to intercept the lookups other users made (even before you answered with a (likely better) solution), but then I realized that my idea was unusable anyway.
The paper I referred to talked about a single entity snooping on the entire dht by creating a gazillion fake nodes that just forward requests, and my idea was to build a Popcorn Time-like thing on top of such a snoop machine.
But then I realized that for everyone that did so it would expand the network a lot, and the previous logarithmic scaleability would become something more linear (since everyone needs to see a fair amount of all lookups to be able to create an index).
The only solution I could think about then was a smaller distributed cluster of supernodes that does this on behalf of the entire network, store their indices redundantly, and reply to search requests. The code for this supernode would be something philantropists would run voluntarily to make the network run more efficiently. Quite a lot more complicated than my original hacky idea.
All other solutions seem to require that the default BT client behavior changes significantly.
For example, allow dht nodes to change id if they get too much or too little responsibility, and thus allow distributing search terms without hashing them and still not completely kill nodes that happen to end up near popular parts of the name space. This to allow a completely distributed search index. It all became too complex :)
And yeah, what a coincidence! Finding you here at github AND at the office makes me hopeful about finding a good 20% to spend my time on :D
Funny you should say that. I spent quite a bit of time working on such a project, across years in fact. I kept it private. It's called "supernode". :-). The domain is "magnets.im".
It watches for infohashes and grabs their torrents using metadata searches. It works, but it's not production quality. It's also how I learned Go (along with this library), so the code isn't really great.
I didn't make it public because at the end of the day the service I'd be providing is very similar to a torrent indexing site. And I wasn't really comfortable with releasing it.
It is a really cool research project, but I stopped working on it - in part because it was a bit lonely. Let me know if you think we should resurrect it :).
I still have one supernode running. Check out the dashboard:
We should move this to a chat of some kind!
I asked yvesj@ a message :)