Hanging database - indexer waiting for transactions, query waiting for indexer
jarohen opened this issue · comments
From @nivekuil on Clojurians
Details:
-
in-mem tx-log, local-disk storage (haven't tried with local/local or in-mem/in-mem)
-
seems to be a necessary condition that it's written at least a block to disk (i.e. starting from a clean tx-log and storage doesn't repro this)
-
then
pmap
50 docs (pmap
seems to be a necessary condition)(pmap (fn [x] (xt/submit-tx node [[:put-docs :test {:xt/id (random-uuid)}]])) (repeat 50 1))
-
further queries then hang - the transaction indexing thread awaiting further transactions from the log, the query awaiting transactions to be indexed.
"xtdb-tx-subscription-pool-1-thread-2" #128 [2042437] prio=5 os_prio=0 cpu=0.15ms elapsed=84.41s tid=0x0000739c0cf185a0 nid=2042437 waiting on condition [0x0000739f0cdcc000] java.lang.Thread.State: WAITING (parking) [...] at java.util.concurrent.Semaphore.acquire(java.base@22/Semaphore.java:318) at xtdb.log.NotifyingSubscriberHandler$fn__74636.invoke(log.clj:106) [...] "nREPL-session-a1361af3-7e3a-4488-abcc-65364b4bbee9" #252 [2044496] daemon prio=5 os_prio=0 cpu=17.34ms elapsed=453.42s tid=0x0000739cc00012f0 nid=2044496 waiting on condition [0x0000739f0e5e3000] java.lang.Thread.State: WAITING (parking) [...] at clojure.core$deref.invoke(core.clj:2323) at xtdb.api$q.invokeStatic(api.clj:159) [...]
-
latest-completed-tx is behind latest-submitted-tx. Seems to be plenty of transactions available through the log - my (unsubstantiated) gut feel is that it's a race condition in the in-memory tx-log transaction notification?
-
This also may well be an issue caused by an in-memory tx-log vs a local-disk storage, with the former being reset without the latter.