multi-threaded dictionary crash
linas opened this issue · comments
Multi-threaded update of the dictionary is not thread-safe. This was always the case. We knew this all along. This didn't matter, when the dictionaries were just file-backed. But now that they can be dynamic objects, it is time to fix this.
I get this:
#3 0x00007ffff7e612fc in malloc_printerr (
str=str@entry=0x7ffff7f85670 "double free or corruption (out)")
at malloc.c:5347
#4 0x00007ffff7e62fa0 in _int_free (av=0x7ffff7fb8b80 <main_arena>,
p=0x7fff300aee10, have_lock=<optimized out>) at malloc.c:4314
#5 0x00007fffecb8cd95 in condesc_grow (ct=0x7fff4c00cfc8)
at ../../link-grammar/connectors.c:539
#6 condesc_add (ct=ct@entry=0x7fff4c00cfc8, constring=0x7fff30091510 "MDB")
at ../../link-grammar/connectors.c:558
#7 0x00007fffecba6d2d in make_connector_node (
dict=dict@entry=0x7fff4c00cea0, mp=<optimized out>,
linktype=0x7fff3a7fac70 "MDB", dir=<optimized out>,
multi=multi@entry=false) at ../../link-grammar/dict-ram/dict-ram.c:530
Above stack trace fixed in #1380 Now I get
Thread 103 "cogserv:eval" received signal SIGSEGV, Segmentation fau
#0 pool_alloc_vec (mp=0x7fff4c0155d0, vecsize=vecsize@entry=1)
at ../../link-grammar/memory-pool.c:189
#1 0x00007fffecbaad75 in pool_alloc (mp=<optimized out>)
at ../../link-grammar/memory-pool.h:116
#2 Exp_create (mp=<optimized out>)
at ../../link-grammar/dict-ram/dict-ram.c:442
#3 make_connector_node (dict=dict@entry=0x7fff4c00ca60, mp=<optimized out>,
linktype=0x7fff22ffbc70 "DPL", dir=<optimized out>,
multi=multi@entry=false) at ../../link-grammar/dict-ram/dict-ram.c:527
Above fixed in #1381 ... Next up,the string-set:
#0 find_place (ss=0x7fff50001040, h=940135772, str=0x7fff6f7f4c70 "GLKDB")
at ../../link-grammar/string-set.c:149
#1 string_set_add (source_string=source_string@entry=0x7fff6f7f4c70 "GLKDB",
ss=0x7fff50001040) at ../../link-grammar/string-set.c:189
#2 0x00007fffecbaaefc in make_connector_node (
dict=dict@entry=0x7fff5000ce90, mp=0x7fff50015be0,
linktype=0x7fff6f7f4c70 "GLKDB", dir=<optimized out>,
multi=multi@entry=false) at ../../link-grammar/dict-ram/dict-ram.c:530
Above fixed in #1383 Next one is this:
#0 sort_condesc_by_uc_constring (dict=dict@entry=0x7fff5000cea0)
at ../../link-grammar/connectors.c:429
#1 0x00007fffecb90f96 in condesc_setup (dict=0x7fff5000cea0)
at ../../link-grammar/connectors.c:590
#2 0x00007fffecb8f715 in sentence_parse (sent=sent@entry=0x7fff10002dd0,
opts=opts@entry=0x7fff10006890) at ../../link-grammar/api.c:692
Above is fixed by #1384
I'm starting to think that many of the above fine-grained locks could be removed and replaced by a more coarse-grained lock in dict-atomese. This is because there are still more races, which need the coarse-grained lock to fix. Which makes the fin-grained locks pointless...
Nope. Still have issues.
#0 sort_condesc_by_uc_constring (dict=0x55598d237940)
at ../../link-grammar/connectors.c:429
#1 condesc_setup (dict=0x55598d237940) at ../../link-grammar/connectors.c:581
#2 0x00007fffe836c735 in sentence_parse (sent=sent@entry=0x7fff100027a0,
opts=opts@entry=0x7fff10001450) at ../../link-grammar/api.c:692
Do you have a full backtrace?
Do you have a full backtrace?
I'm working on a redesign to take care of this..
The redesign will notify the dictionary when parsing starts and stops. This will allow the dictionary to know what the words are in the current sentence, and do special per-sentence stuff.
The redesign that (I hope) fixes the last stack trace was #1394 ... all the earlier pull reqs were cleanup to enable this change. I'm hoping this is the last one...
As of commit f52ac71, there is a strange parsing problem, that doesn't seem related to the deduplicate_linkages() bugs I pointed out in #1378: Many sentences silently get ignored (no parsing and no error).
E.g:
He said that, finding that it was impossible to get work as a waiter, he would work as a janitor
I don't have any idea of a possible reason (haven't bisected yet), but it seems it depends on the number of words - if you remove one by one from the end you finally get a response, but then if you add any random word then you get nothing. I'm still investigating the apparent problems in removing duplicate parses, so for now I'm just reporting this error without looking deeply into it.
BTW, there is also seems to be a performance hit (+~10% on short sentences, +~7% on fix-long, +6%% on pandp-union). I wonder why, as you have removed the fine-grain locks (I will try to profile it latter).
Thank you. It's no longer crashing, but it is behaving very oddly. For me, the vast majority of sentences seem to have gone AWOL. But I'm not using the command line tool, so its harder for me to debug.
One obvious culprit is #1385 -- it actually changes things.
The other is to revert #1378 until such time that the several issues you mention are fixed. ...
what a mess. "move fast and break things".
The dedupe is the cause of the missing linkages, and I've got a fix for that.
OK, once again -- I think everything is now fixed, and all commentary has been handled. You mentioned slowdowns ... I too measure things and keep poor notes.
The only slowdown I saw was from #1103 but my notes are not clear it that was recovered or not. I've been observing an 8% slowdown consistently, ever since #1103 But this could also be explained by other changes, too.