ExclusiveAcqFailed(BorrowMutError) when triggering read transaction in observe callback

[Stefan Wehrmeyer]

If you observe changes on a document or a type via callbacks, the callbacks run when the transaction is committed. When the callback tries to perform any operation that requires a (read) transaction it will panic with an ExclusiveAcqFailed(BorrowMutError). This is the case in v0.7.0a1 with new internal transaction handling and a regression to v0.6.

Python test case

def test_callback_after_apply_update():
    # setup to get an update
    remote_doc = Y.YDoc()
    text = remote_doc.get_text("test")
    with remote_doc.begin_transaction() as txn:
        text.extend(txn, "Hello")
    update = Y.encode_state_as_update(remote_doc)

    doc = Y.YDoc()
    text = doc.get_text("test")

    target = None
    nodes = None

    def callback(e):
        nonlocal target
        nonlocal nodes
        target = e.target
        nodes = e.delta
        # Crash occurs when `__str__` is called on target which requires a `ReadTxn`
        print("callback", target, nodes)

    subscription_id = text.observe(callback)
    Y.apply_update(doc, update)
    print("after update", target, nodes)
    text.unobserve(subscription_id)

The callback is executed during the call to commit when TransactionMut has not been dropped yet (commit is possibly called from drop), so trying to acquire another transaction in order to call e.g. __str__ on a type fails. At least that is my interpretation of the resulting stack trace.

A workaround is to use the callback to keep references and then use them after the callback and the commit has returned.

yrs passes the committed transaction to observer callbacks and those can be used in the callback as ReadTxn – even though the transaction is already committed (ReadTxn is as far as I can tell just a borrow check and not actually used).

I'm wondering if there's a way to fix this? We could try to store the callbacks and call them after the transaction is completed.

These are the relationships between the different objects in v0.7.0a1:

flowchart TB
    YTransactionInner -- ManuallyDrop --> TransactionMut[[TransactionMut]]
    YTransaction -- Rc RefCell --> YTransactionInner
    YDocInner --> Doc[[Doc]]
    YDocInner -- Option Weak RefCell --> YTransactionInner
    YDoc -- Rc Refcell --> YDocInner

Loading

[David Brochart]

But what do we want to do with the target in the callback anyway? We cannot mutate it, so wouldn't it be better to directly return its representation (string for a YText, list for a YArray, etc.)?
Otherwise, if we want to react to changes with other changes, since we cannot mutate the target in the callback, we need to schedule another callback to be called later. That could be done if we are running in an event loop, by running a task:

async def other_callback(target):
    target.extend(", World!")

def callback(e):
    asyncio.create_task(other_callback(e.target))