Neo4j 2.0
atomos opened this issue · comments
Hi,
I'm just starting out using Neo4j and I'd like to use 2.0 (I have 2.0.1 community installed). I see that neo4django was only tested against neo4j 1.8.2-1.9.4, but have people gotten it working with 2.x? I installed the gremlin plugin but can't create or query through neo4django.
create:
In [8]: NeoProfile.objects.create(profile_id=1234)
[INFO] requests.packages.urllib3.connectionpool#214: Resetting dropped connection: localhost
---------------------------------------------------------------------------
StatusException Traceback (most recent call last)
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <module>()
----> 1 NeoProfile.objects.create(profile_id=1234)
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/manager.pyc in create(self, **kwargs)
41
42 def create(self, **kwargs):
---> 43 return self.get_query_set().create(**kwargs)
44
45 def filter(self, *args, **kwargs):
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in create(self, **kwargs)
1295 if 'id' in kwargs or 'pk' in kwargs:
1296 raise FieldError("Neo4j doesn't allow node ids to be assigned.")
-> 1297 return super(NodeQuerySet, self).create(**kwargs)
1298
1299 #TODO would be awesome if this were transactional
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/query.pyc in create(self, **kwargs)
375 obj = self.model(**kwargs)
376 self._for_write = True
--> 377 obj.save(force_insert=True, using=self.db)
378 return obj
379
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in save(self, using, **kwargs)
315
316 def save(self, using=DEFAULT_DB_ALIAS, **kwargs):
--> 317 return super(NodeModel, self).save(using=using, **kwargs)
318
319 @alters_data
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/base.pyc in save(self, force_insert, force_update, using)
461 if force_insert and force_update:
462 raise ValueError("Cannot force both insert and updating in model saving.")
--> 463 self.save_base(using=using, force_insert=force_insert, force_update=force_update)
464
465 save.alters_data = True
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in save_base(self, raw, cls, origin, force_insert, force_update, using, *args, **kwargs)
331
332 is_new = self.id is None
--> 333 self._save_neo4j_node(using)
334 self._save_properties(self, self.__node, is_new)
335 self._save_neo4j_relationships(self, self.__node)
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in _save_neo4j_node(self, using)
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in trans_method(func, *args, **kw)
95 #TODO this is where generalized transaction support will go,
96 #when it's ready in neo4jrestclient
---> 97 ret = func(*args, **kw)
98 #tx.commit()
99 return ret
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in _save_neo4j_node(self, using)
359 self.__node = conn.gremlin_tx(script, types=type_hier_props,
360 indexName=self.index_name(),
--> 361 typesToIndex=type_names_to_index)
362 return self.__node
363
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/neo4jclient.pyc in gremlin_tx(self, script, **params)
177 will be wrapped in a transaction.
178 """
--> 179 return self.gremlin(script, tx=True, **params)
180
181 def cypher(self, query, **params):
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/neo4jclient.pyc in gremlin(self, script, tx, raw, **params)
166 try:
167 return send_script(include_unloaded_libraries(lib_script),
--> 168 params)
169 except LibraryCouldNotLoad:
170 if i == 0:
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/neo4jclient.pyc in send_script(s, params)
151 if raw:
152 execute_kwargs['returns'] = RETURNS_RAW
--> 153 script_rv = ext.execute_script(s, params=params, **execute_kwargs)
154 if isinstance(script_rv, basestring):
155 if LIBRARY_ERROR_REGEX.match(script_rv):
/Users/atomos/workspace/Project-Vitamin/src/neo4j-rest-client/neo4jrestclient/client.py in __call__(self, *args, **kwargs)
2313 except (ValueError, AttributeError, KeyError, TypeError):
2314 pass
-> 2315 raise StatusException(response.status_code, msg)
2316
2317 def __repr__(self):
StatusException: Code [400]: Bad Request. Bad request syntax or unsupported method.
Invalid data sent: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: groovy.lang.MissingMethodException.setMaxBufferSize() is applicable for argument types: () values: []
query:
In [9]: NeoProfile.objects.filter(profile_id=1234)
Out[9]: ---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <module>()
----> 1 NeoProfile.objects.filter(profile_id=1234)
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
236 self.start_displayhook()
237 self.write_output_prompt()
--> 238 format_dict = self.compute_format_data(result)
239 self.write_format_data(format_dict)
240 self.update_user_ns(result)
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/displayhook.pyc in compute_format_data(self, result)
148 MIME type representation of the object.
149 """
--> 150 return self.shell.display_formatter.format(result)
151
152 def write_format_data(self, format_dict):
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/formatters.pyc in format(self, obj, include, exclude)
124 continue
125 try:
--> 126 data = formatter(obj)
127 except:
128 # FIXME: log the exception
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
445 type_pprinters=self.type_printers,
446 deferred_pprinters=self.deferred_printers)
--> 447 printer.pretty(obj)
448 printer.flush()
449 return stream.getvalue()
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
358 if callable(meth):
359 return meth(obj, self, cycle)
--> 360 return _default_pprint(obj, self, cycle)
361 finally:
362 self.end_group()
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
478 if getattr(klass, '__repr__', None) not in _baseclass_reprs:
479 # A user-provided repr.
--> 480 p.text(repr(obj))
481 return
482 p.begin_group(1, '<')
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/query.pyc in __repr__(self)
70
71 def __repr__(self):
---> 72 data = list(self[:REPR_OUTPUT_SIZE + 1])
73 if len(data) > REPR_OUTPUT_SIZE:
74 data[-1] = "...(remaining elements truncated)..."
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/query.pyc in __len__(self)
85 self._result_cache = list(self.iterator())
86 elif self._iter:
---> 87 self._result_cache.extend(self._iter)
88 if self._prefetch_related_lookups and not self._prefetch_done:
89 self._prefetch_related_objects()
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in iterator(self)
1274 using = self.db
1275 if not self.query.can_filter():
-> 1276 for model in self.query.execute(using):
1277 yield model
1278 else:
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in execute(self, using)
1161 conn = connections[using]
1162
-> 1163 groovy, params = self.as_groovy(using)
1164
1165 raw_result_set = conn.gremlin_tx(groovy, **params) if groovy is not None else []
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in as_groovy(self, using)
925 # add the typeNodeId param, either for type verification or initial
926 # type tree traversal
--> 927 cypher_params['typeNodeId'] = self.model._type_node(using).id
928
929 type_restriction_expr = """
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in _type_node(cls, using)
411 return cls.__type_node_memoized(using)
412 else:
--> 413 return cls.__type_node_classmethod(using)
414
415 @classmethod
/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in __type_node(cls, using)
394 script_rv = conn.gremlin_tx(script, types=type_hier_props)
395 except Exception, e:
--> 396 raise RuntimeError(error_message, e)
397 if not hasattr(script_rv, 'properties'):
398 raise RuntimeError(error_message + '\n\n%s' % script_rv)
RuntimeError: ('The type node for class NeoProfile could not be created in the database.', StatusException())
My model is incredibly complex:
class NeoProfile(neomodels.NodeModel):
profile_id = neomodels.IntegerProperty(indexed=True)
I'm also trying to upgrade to Neo4j 2. Now that the Gremlin plugin is installed, I get the same error that you end up with (from the API):
{
error_message: "('The type node for class Word could not be created in the database.', StatusException())",
traceback: "Traceback (most recent call last): File "/opt/phaidra/env/src/django-tastypie/tastypie/resources.py", line 195, in wrapper response = callback(request, *args, **kwargs) File "/opt/phaidra/env/src/django-tastypie/tastypie/resources.py", line 426, in dispatch_list return self.dispatch('list', request, **kwargs) File "/opt/phaidra/env/src/django-tastypie/tastypie/resources.py", line 458, in dispatch response = method(request, **kwargs) File "/opt/phaidra/api/api.py", line 421, in get_list if len(words) < 1: File "/opt/phaidra/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 106, in __len__ self._result_cache = list(self.iterator()) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/query.py", line 1283, in iterator piece = list(clone.execute(using)) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/query.py", line 1162, in execute groovy, params = self.as_groovy(using) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/query.py", line 926, in as_groovy cypher_params['typeNodeId'] = self.model._type_node(using).id File "/opt/phaidra/env/src/neo4django/neo4django/db/models/base.py", line 411, in _type_node return cls.__type_node_classmethod(using) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/base.py", line 394, in __type_node raise RuntimeError(error_message, e) RuntimeError: ('The type node for class Word could not be created in the database.', StatusException()) "
}
Namely:
RuntimeError: ('The type node for class Word could not be created in the database.', StatusException()
Have you made any progress on this?
Sorry, I gave up on neo4django. I don't remember seeing that error.
I'll be working on getting some progress here, ill post any notes as i go.
I think it wont be too difficult since the library were using to talk to neo4j has been working on 2.0 for 6 months now. We may just need to use the ParserVersion feature to inform neo4j that were going to query it with 1.8 or 1.9 query semantics. (at least i hope its that easy lol)
Ive got some progress on 2.0 and 2.1 in my fork here: techdragon@fac398b
Ive been chipping away at the errors that caused the issues. After updating the neo4jrestclient library, adding detection code to catch when the version of neo4j is 2.0 or higher, and then creating a secondary groovy script to be called when we we are on 2.0 or higher in order to be able to try and develop something compatible with the old way as well as 2.0... I have been able to steadily comment my way through the groovy script and pinpoint a few of the problems with gettingneo4django working on Neo4j 2.x with versions of the Gremlin Plugin for 2.x ... All of which are in the groovy script code.
@mhluongo You're the only person thats ever touched the groovy code from the look of it, so while I'm sure ill eventually work it out, this may be much more efficient if you look at the issue. It appears that in order to be compatible with neo4j 2.x we need to remove all the transaction handling code and make some changes to how it queries and creates nodes. Like the changes and comments in the following links:
dbunskoek/bulbs@02c861b
https://groups.google.com/forum/#!topic/gremlin-users/OUkpukj8YYY
https://github.com/tinkerpop/blueprints/blob/master/doc/The-Major-Differences-Between-Blueprints-1.x-and-2.x.textile
I've tried to work out some of this groovy code stuff myself but it seems from your blog post you may be doing some advanced groovy code magic ( http://mattluongo.com/post/client-side-gremlin-libraries-in-neo4j ) so while I'll keep working on this, I'm pretty sure you may be able to either point me in the right direction so i can finish fixing my fork for a PR, or post some code i can use that will help.
edit:
It also seems to be related to the "type node" behaviour neo4django appears to either implicitly depend on or rely on as part of its use of gremlin
edit2:
I think you may have already logged an issue for this... hard to tell without a description ;-) #170
edit3:
I've tracked it back down to the assumption of neo4j having a node(0) "reference node" and the lack of the reference node is causing all kinds of trouble. Since the reference node assumption is a major roadblock on my ability to further tinker with the gremlin pathway, incase this 1.x reference node assumption proves to be a major hurdle to overcome, I'm going to take a shot at building a pure cypher implementation on top of the new GSOC 2014 django meta API in trunk (as mentioned here #252) and try to keep its schema as close to the one provided by neo4django as i can.
@techdragon thanks for your work so far! I've done some client work where I used a similar approach (Groovy + Gremlin + Cypher) against Neo4j 2.0+. I'll check if that's applicable.
Heh, yeah, that's the issue. It's actually pretty easy to just create a "reference node", and always use that by pulling from an index, instead of relying on the old node(0) business. I think that'd be the quickest fix. That said, actually using Neo4j 2.0 types instead of the neo4django-constructed type tree would be sweet.
I'd caution you on going all-Cypher- Cypher is great, but there are some very basic things it doesn't yet support that neo4django has had to work around with Gremlin / client-side index craziness. It'd be a shame to no longer have the complex filtering done against the indices first.
@mhluongo I'm keeping an eye on this quite closely as my current app progresses so hopefully some of that client work code proves fruitful.
From what I've been building so far, the type/label system is quite powerful and does make creating node types a lot more strait forward.
I'm holding back on starting work on the all cypher neo4django option at the moment for a couple of reasons, one of which is the still evolving state for the new django meta api improvements, the other of which is a general uncertainty as to wether to drop down to direct communication to/from neo4j or not, so my current project is being built using neo4j 2.1 format raw cypher constructed using the helpful https://github.com/bruth/cypher tool and (at the moment) py2neo. I'm taking full advantage of the ability to just "remodel the schema as i go" and letting things evolve for a while before trying to build any kind of tool with it as a primary use case.