Storing custom EJSON in mongodb only sort of works
aldeed opened this issue · comments
From what I can tell, the only reason mongodb throws a not okForStorage error when attempting to insert a custom EJSON type is because the keys begin with "$". But I don't see any good reason why Meteor needs to use "$" instead of any other unusual character. If these lines are changed to use !type and !value, for example, mongo does not have a problem. I think some other random places in the codebase would need changes, too, but not many.
If you change all outgoing code to use "!" and change all incoming code to recognize "!" or "$", this would be backwards compatible.
Hi @aldeed. I think this should work. There is code in mongo-livedata to translate. See https://github.com/meteor/meteor/blob/devel/packages/mongo-livedata/mongo_driver.js#L51
@n1mmy, if it should work, then why do the docs say this:
MongoDB cannot store most user-defined types natively on the server. Your type will work in Minimongo, and you can send it to the client using a custom publisher, but MongoDB can only store the types defined in BSON.
And why does mongodb throw an error when I try it?
Yeah, that paragraph should have been removed from the docs, thanks!
re "why does mongodb throw an error", I have no idea because you haven't given me a reproduction. https://github.com/meteor/meteor/wiki/Contributing-to-Meteor#reporting-a-bug-in-meteor
Sorry, didn't include a repro because I didn't realize I was reporting a bug. Here:
https://github.com/aldeed/ejson-test
Just start up the project and you will see the error logged to terminal. It's using the ejson custom type example from the eventedmind episode. I initially saw the error with a different custom type, but I figured I'd copy/paste from the episode just to make sure there was nothing weird about my custom type. Still the same error.
And just in case you don't get an error, this is what is logged for me:
I20140313-21:50:04.638(-5)? { [MongoError: not okForStorage]
I20140313-21:50:04.822(-5)? name: 'MongoError',
I20140313-21:50:04.823(-5)? err: 'not okForStorage',
I20140313-21:50:04.823(-5)? code: 12527,
I20140313-21:50:04.823(-5)? n: 0,
I20140313-21:50:04.823(-5)? lastOp: { _bsontype: 'Timestamp', low_: 0, high_: 0 },
I20140313-21:50:04.823(-5)? connectionId: 9,
I20140313-21:50:04.823(-5)? ok: 1 }
Thanks for the reproduction. Your repro is very similar to the tests in Meteor that show that this "works", and comparing the two carefully made it clear what the difference was. sigh, this feature basically can't work as designed. I'm going to revert the docs change I pushed tonight and continue to document this as not working. We'll think more of this pre-1.0 as we tighten up the DDP spec.
Notably, this particular MongoDB check doesn't appear to occur inside insert, or inside an update which doesn't increase the size of the document? But it does for an update that does increase the size of the document? Ah, MongoDB.
Weird. I'm glad I'm not crazy. Thanks for taking a look!
cc: @raix
Again, ejson should only fill the ago between ejson and bjson. I like the idea of custom ejson, but it's overhead and should prop. Be a seperate thing/package?
It looks like the inconsistent behavior of "fields that have a $ in the middle" between different Mongo commands was considering a bug and will be fixed in the currently-in-rc 2.6: mongodb/mongo@4211c6a
Given that you appear to be the first person to notice that this feature only sort of works in the year since it was implemented, I think it's probably OK to say that the solution is "wait until Mongo 2.6 is stable and use it".
I agree that "wait for Mongo 2.6" is fine. Thanks!
@n1mmy @glasser, I just wanted to point out that it looks like this fix is in mongo 2.5.2+. We're starting to get a lot of questions about this bug in relation to the collectionFS package. (Particularly, lots of complaints that $push works with custom ejson but $set does not.) Is there an ETA for when Meteor will ship with 2.5.2+? Are there any known issues with running against 2.5.2?
"get Meteor supporting Mongo 2.6" is on my personal todo list, though it may not happen (unless somebody else beats me to it) until after my honeymoon, so probably October.
With meteor 1.1 supporting Mongo 3.0 & 2.6 is this considered fixed?
Yes, this seems to work as of Meteor 1.0.4 (or well, any use of Mongo 2.6 or newer).