A frontend (possibly database agnostic) API for ShareJS, and other live database implementations
mizzao opened this issue · comments
I've been following the redevelopment of ShareJS 0.7 and livedb, and I've been watching with interest because of how I've been using ShareJS. Specifically, I wanted to throw a crazy idea out there, just for the record:
Suppose that there was more than one way to implement a live database that can accept and push updates to/from multiple clients. In particular, suppose that it was easy to give clients access to document revisions and snapshots in browser memory. Could it ever make sense to separate the client-side API for ShareJS with the API for the live database, so that different live backends could implement the same operations?
The reason I ask is because, as you may or may not know, Meteor's mongo-livedata package, which is one of its core features, has much of the same functionality as livedb, except arguably in a more generalized (and more widely tested) way. As I've been the main proponent of enabling Meteor apps to use ShareJS, it's been on the back of my mind whether things could be more integrated in the future. Currently, my approach is to start a separate ShareJS middleware stack alongside Meteor, and send along all the ShareJS ops independently of other Meteor data. Both stacks are actually reading/writing to the same Mongo database. In the future, this would probably result in two versions of live mongo-backed databases running alongside each other. An alternative, cleaner approach would be to work directly off Meteor's synced collections with some smart publication logic and a client-only API.
I know that @josephg works for Derby now, and they probably see Meteor as a competitor. However, it pains me to see the work going into livedb at the same time as Meteor's live data, and the duplication of effort on open source software. Hence, as a related question, I'm just wondering if any collaboration is possible, and pointing it out just in case.
As far as I know livedb in Meteor pings Mongo every 10 seconds. And if you want to scale it (multi-process), you will have 10s lags for data synchronization. Also there is no conflict resolution.
Also as far as I know there is no way to change Meteor livedb to something that use OT (ShareJS for example), because Meteor DDP-protocol uses web-sockets, which can not guarantee message order and that is critical for OT.
So the only way for using ShareJS with Meteor is to wrap it in a smart package and use it along side with Meteor livedb.
It looks a little bit confusing to have two livedbs in one project, just because one of them is limited. Maybe it`s better to switch from Meteor to Derby/Racer/ShareJS? What are the arguments for Meteor?
@vmakhaev I think you may have some misinformation about Meteor. Meteor only pings Mongo every 10 seconds for direct writes to Mongo outside of the Meteor app. Any writes from DDP or a client are instant and pushed out to all other subscribers. Meteor's mongo implementation is now being rewritten as a Mongo replica set secondary so this external lag will be eliminated. This is about as close integration with Mongo as one can get.
From your explanation, are you saying livedb is going to replace the synchronization db that Racer is using right now?
In regard to the message ordering, there is no need for the websocket to guarantee the message order; the server has the last word on the ordering of messages received and can publish the revisions in some canonical order, and mongo provides indexing and whatnot.
Finally, I don't want to get into an argument about Meteor vs Derby right now - I realize that ShareJS users are likely to be firmly in Derby-land, but I'm trying to bridge the gap. When I made the choice between Meteor and Derby, my main reasons were due to more users (and hence a bigger community), more rapid development, and better documentation.
Agreed that GitHub issues are not the best place for a Meteor vs. Derby discussion.
The original issue asked whether it would make sense to use the LiveDB implementation in npm (which is what ShareJS uses as a backend) to power other frameworks, such as Meteor.
Our teams follow each others' efforts closely, and we have learned a lot from each other in the past two years. We know each other and share insights occasionally. However, we have very different goals, and it makes more sense to work independently.
From a technical perspective, these two systems are very different and have very different tradeoffs. ShareJS requires an operation log, which means that directly manipulating data in the Mongo database is not allowed. Meteor puts a lot of effort into making it simple to get up and running, so they would not want to add a requirement like that to their primary implementation. ShareJS can't use the Meteor backend stack, because it doesn't an operation log. In addition, we currently have to use an external system like Redis (FoundationDB could also work) to actually power the transformation, commit and PubSub process, because MongoDB doesn't have sufficient capabilities alone.
In your case of already running ShareJS and Meteor server at the same time, you could build a Meteor package that would sit on top of the npm LiveDB and translate to DDP to work with Meteor clients. However, I don't see the benefit, so I wouldn't recommend it.
I'm going to close this, because it isn't something that it makes sense for ShareJS to support directly.
@mizzao's idea of implementing the livedb API in the browser is an interesting one all the same. It would be much simpler than what we have now in some ways, but the query support either wouldn't work very well or it would need to be extremely chatty to do the same thing. For example, right now the client can say "find me all documents where x=5 and subscribe me to them". Thats not an API that livedb provides - the sharejs server implements that by first doing the query through livedb then bulk subscribing to resulting documents. If livedb API was moved to the client, it would add an extra round-trip there.
For regular document subscriptions and stuff though it might be doable, and it might be a cleaner wire API. (Though its also not so far from what we have going over the wire now)
@josephg what you just described seems pretty easy to do with Meteor's publish/subscribe collection model, and it's what I imagined when I first posted this. The server's document publication would take either a single document id or a query ("find all documents where x=5") and return a set of document cursors in the subscription. Each cursor contains the latest snapshot of the document plus all revisions from there up to the most recent one. The document is edited simply by inserting another revision on the client, which propagates to the server and is assigned a proper order. Because the subscriptions are all live, the list of revisions is pushed out to all other clients subscribed to the same document as well. In this case the conflict resolution wouldn't even be all that complicated because the server has the last word on the total ordering of revisions, and this can implicitly be handled by an index in the database.
I am not that familiar with the implementation of livedb, but it seems like you would need to do pretty much the same thing to accomplish this. Besides, other types of subscriptions would be possible, such as the for the entire history, allowing the client to replay and view the document since the first revision.
I think you understand already, but what I'm imagining is a client side API where edits can just happen using much simpler commands. This can either talk directly to the ShareJS stack on the server, network and all, or just pass through to a client-side live database API which can both submit revisions and receive pushes of new data. In Meteor's case, the live data pub/sub model supports general collections of JSON documents, so it could handle not only OT-type data but other queries as well.
Just wanted to check in about this: is ShareJS 0.7 and future versions going to be married to Derby, or will the API be abstracted enough to allow for other server-client transports?
I've done a decent job of integrating the ShareJS 0.6 stack with Meteor, along with the Ace editor, but a more robust future implementation would be to implement OT using Meteor's DDP protocol (https://trello.com/c/tkBErvIk/39-operational-transformation), which doesn't seem all that far-fetched given my understanding of the ShareJS database structure. I'm just asking around about this right now to keep tabs on future options.
cc @gadicc
ShareJS isn't married to derby. Its a completely standalone library (it depends on livedb, but thats my library too). ShareJS is still a library that you can add in to your app to selectively add live editing capabilities. It just also happens to implement a lot of features you need for dealing with databases as well (queries, projections, bulk fetches, etc) because derby needs those features to work well. I don't know much about meteor internals, but I'd be happy to chat about design decisions if you want.
Great to hear, @josephg. It appears to me that Derby has had some difficulty gaining traction recently, and I don't want your valuable work to be lost. As I mentioned in the Trello post, livedb has a lot of similar features to Meteor's livedata, and it would be great for you to at least be aware of what's been done there so that we don't duplicate any large efforts unnecessarily. In the best case, we might be able to merge the projection/aggregation (?) features of livedb with the robustness of livedata, and produce a really awesome client-server database API.
We are big on modularity, and we have no intention of tying ShareJS to the
development of Derby, Meteor, or any other framework. If you want to
integrate ShareJS with Meteor or use them side-by-side, it should work just
fine.
I think the fact that ShareJS has additional features like query
subscriptions and multi-process/server realtime PubSub to help support
Racer/Derby's use cases will ease integration with Meteor if anything. We
are happy to offer advice on the implementation.
I've decided not to promote Derby too much over the past couple years as
we've matured the framework. I will start to promote it as soon as the
upcoming release of 0.6 is complete. But regardless, we support ShareJS as
its own project and are happy to see it succeed independently.
On Wed, May 14, 2014 at 4:52 PM, Andrew Mao notifications@github.comwrote:
Great to hear, @josephg https://github.com/josephg. It appears to me
that Derby has had some difficulty gaining traction recently, and I don't
want your valuable work to be lost. As I mentioned in the Trello post,
livedb has a lot of similar features to Meteor's livedata, and it would be
great for you to at least be aware of what's been done there so that we
don't duplicate any large efforts unnecessarily. In the best case, we might
be able to merge the projection/aggregation (?) features of livedb with the
robustness of livedata, and produce a really awesome client-server database
API.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/277#issuecomment-43154159
.
@nateps my original suggestion was whether the ShareJS code is sufficiently modular so that the client code could implement an API that would allow for a different live database stack to sync with the server. I'm fairly confident that most of the livedb operations could be implemented over Meteor's DDP. This would allow us to avoid the side-by-side network stacks in most use cases.
However, I'm happy to keep going with the current approach if that's what we have to do - just wanted to check and see if there was any common work that we could build upon.