RepresenterError: cannot represent an object: OrderedDict(…)
Karmak23 opened this issue · comments
Hi,
Sorry to ask what is a question and probably not a bug here, but I could not find any clue after hours of searching (and even upgrading python, see below). I've got a weird problem on one machine:
Could not import <DeserializedObject: core.ChainedItem(pk=16)>
Traceback (most recent call last):
File "/home/1flow/www/src/oneflow/core/models/reldb/processor/loader.py", line 340, in load_processors_from_latest_fixture
obj.save()
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/core/serializers/base.py", line 164, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/base.py", line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/base.py", line 635, in _save_table
forced_update)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/base.py", line 679, in _do_update
return filtered._update(values) > 0
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/query.py", line 510, in _update
return query.get_compiler(self.db).execute_sql(None)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 980, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 776, in execute_sql
sql, params = self.as_sql()
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 945, in as_sql
val = field.get_db_prep_save(val, connection=self.connection)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/yamlfield/fields.py", line 38, in get_db_prep_save
default_flow_style=False
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/yaml/__init__.py", line 202, in dump
return dump_all([data], stream, Dumper=Dumper, **kwds)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all
dumper.represent(data)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/yaml/representer.py", line 28, in represent
node = self.represent_data(data)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/yaml/representer.py", line 67, in represent_data
node = self.yaml_representers[None](self, data)
File "/home/1flow/.virtualenvs/1flow/lib/python2.7/site-packages/yaml/representer.py", line 247, in represent_undefined
raise RepresenterError("cannot represent an object: %s" % data)
RepresenterError: cannot represent an object: OrderedDict([('prepare', OrderedDict([('force_content_type', 'CLEANED_HTML'), ('use_excerpt_as_content', True)])), ('process_metadata', False)])
This happens on an Ubuntu 12.04 server with python 2.7.3 (but also with 2.7.9) and latest yamlfield package, when trying to load a JSON fixture that contains a YAML field.
The weird issue is that it doesn't crash (the fixture loads cleanly) on my Arch Linux desktop (also Python 2.7.9 and same yamlfield), where the fixture file was generated.
At some point I thought it could come from the bugs of OrderedDict in python 2.7.3 which are fixed in python 2.7.7 (thus I upgraded to 2.7.9 to be sure I have the same python version on both sides), but it seems not to make a difference.
Both machines [should] have the same configuration a priori, same version of all requirements. I've reinstalled a full virtualenv with no luck on the failing machine. But obviously, there is something different, and I can't find what.
I'd like some advise on what to search for next, because I've no idea left on where this problem can come from. Can the version of PostgreSQL matter at that point ?
thanks for your time & best regards,
Hmm. Very interesting. An answer does not leap to mind but I'll take a look at it later when I have some time. I guess this is what I get for trying to add the "feature" of OrderedDicts. ;)
Since earlier versions of the package don't use an OrderedDict, you could try downgrading our package for now to see if that gets you by.
I'll try to downgrade, but I fear this will fail either, because the JSON holds an ordereddict dependant YAML representation (the parameters item) :
…
{
"pk": 11,
"model": "core.chaineditem",
"fields": {
"chain": [
"1fs-article-default-forced-utf8"
],
"parameters": "OrderedDict([('download', OrderedDict([('force_encoding', 'utf-8')]))])",
"notes_fr": "…",
"is_active": true,
"check_error": null,
"item_type": [
"core",
"processingchain"
],
"is_valid": true,
"notes_nt": "",
"item_id": 2,
"position": 0,
"notes_en": ""
}
},
…
Perhaps you mean downgrading on the machine that generates the JSON ? I'll try that also. I didn't check how previous (successfully loaded) versions were generated. Thanks for the tip, I'll report in a few hours.
A wild guess (didn't try to downgrade yet): everything was fine with previous non-OrderedDict versions (search for parameters):
{
"pk": 1,
"model": "core.processor",
"fields": {
"rght": 2,
"maintainer": null,
"duplicate_of": null,
"lft": 1,
"description_fr": "",
"requirements": "",
"short_description_nt": "",
"accept_code": "",
"description_nt": "",
"languages": [],
"tree_id": 1,
"short_description_fr": "",
"source_address": "",
"parent": null,
"is_active": true,
"description_en": "This is a 1flow legacy default basic processor.",
"user": [
"karmak23"
],
"slug": "1fs-simple-html-downloader",
"categories": [
1,
6
],
"name": "Simple HTML downloader",
"short_description_en": "",
"parameters": "{'download': {'force_encoding': {'help_en': 'can be \"meta\", \"deep\" or any valid encoding name like \"utf-8\", \"latin9\", etc. See https://github.com/1flow/1flow/wiki/Processing-parameters#simple-html-downloader for details.', 'required': False, 'type': 'string', 'default_en': 'nothing; no encoding forced.'}, 'user_agent': {'help_en': 'a classic user agent string, as seen in any modern web browser.', 'required': False, 'type': 'string', 'default_en': 'Application settings / configuration value'}}}",
"level": 0,
"duplicate_status": null,
"process_code": ""
}
},
3 days ago I upgraded all my requirements, including yamlfield. As previous versions of the fixtures worked, this seems to confirm the OrderedDict implementation could have a problem. But still, I'm wondering why it loads correctly on my local dev machine.
I confirm that moving back to 1.0.0 gives no problem at all anymore. Too bad the OrderedDict implementation produces this awkward bug, it seemed promising. I let the issue open for others to know.
Note: if it had been for my own code, a such deep implementation change would have triggered a 1.1 or a 2.0 release numbering, and probably a changelog note in the README. With 1.0.1, I expected just a bugfix. A growing number of developpers use semantic versionning to avoid such surprises. I usually pin dependancies like django-yamlfield>=1.0.0,<=1.1 to automatically pull fixes, but not updates or API changes without doing it explicitely.
best regards and thanks for the hint, this was exactly what I needed.
A good point. I didn't anticipate this would be a serious issue. We might have to roll it back. Frustrating stuff.
I developed a unit test to try to reproduce your error. I started with the JSON dict you posted here.
It passed with flying colors, much to my surprise. Take a look and let me know what's off.
I think there is a deeper problem I cannot find, because everything worked fine on my development machine, even before the downgrade. As I said, I cannot find the difference between the 2 machines, because on the failing one I even reinstalled python 2.7.9 and a full virtualenv, and it still failed.
Just go on with your test as it is, I will investigate when I get time.