Giters

y-crdt / ypy

Python bindings to y-crdt

Home Page:https://ypy.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Error applying update

davidbrochart opened this issue · comments

commented

I have this error:

"thread '' panicked at 'called Option::unwrap() on a None value', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/yrs-0.7.1/src/transaction.rs:571:59"

when applying this update to a document:

b"\x01\x0e\xde\xd6\xe0\xbe\x0c\t\xc1\xaa\xb7\xc3\xa8\x04s\xaa\xb7\xc3\xa8\x04z\x01\x00\x06\xc1\xde\xd6\xe0\xbe\x0c\t\xaa\xb7\xc3\xa8\x04z\x01\x00\x06\xc7\xc0\xcb\xf4\xe6\r\x96\x01\xab\xf3\xab\xff\r\n\x01(\x00\xde\xd6\xe0\xbe\x0c\x17\x02id\x01w$a584a3d2-2490-4926-b156-0e9fa9b5d718(\x00\xde\xd6\xe0\xbe\x0c\x17\tcell_type\x01w\x04code'\x00\xde\xd6\xe0\xbe\x0c\x17\x06source\x02\x04\x00\xde\xd6\xe0\xbe\x0c\x1a\x052 + 2'\x00\xde\xd6\xe0\xbe\x0c\x17\x08metadata\x01(\x00\xde\xd6\xe0\xbe\x0c \x07trusted\x01x(\x00\xde\xd6\xe0\xbe\x0c\x17\x0fexecution_count\x01}\x0c'\x00\xde\xd6\xe0\xbe\x0c\x17\x07outputs\x00\x08\x00\xde\xd6\xe0\xbe\x0c#\x01v\x04\x0fexecution_count}\x0c\x0boutput_typew\x0eexecute_result\x04datav\x01\ntext/plainw\x014\x08metadatav\x00\x1c\xf6\xe7\x84\x95\x0f\x01\x00\x05\xd4\x80\x96\xbb\x02\x01\x00\r\xa0\x84\x82\xd9\t\x01\x00?\xf9\xa3\xae\xba\x06\x01\x00\x1f\xce\xeb\x90\xaf\x0b\x01\x00\x03\xab\xf3\xab\xff\r\x02\x00\xce\x01\xd5\x01B\xff\xc6\xe3\xe8\x0b\x01\x00\x03\x90\x8b\x8d\xbe\x07\x01\x00\x03\x82\xc0\xd5\xa1\n\x01\x00^\xd0\xa5\xa2\xa7\x02\x01\x00\x03\xfc\xbc\x9a\xa9\x03\x01\x00\x03\xed\xbf\xb4\xd5\x0b\x02\x00\x07\tm\xaa\xb7\xc3\xa8\x04\x02\x00elF\x8c\xe0\xcc\xcc\x03\x01\x00\x95\x01\xc1\x80\xa5\xa4\x07\x02\x00\x03\x17\x92\x02\x9f\xdc\xc6\xa8\x01\x01\x00\x03\xc0\xcb\xf4\xe6\r\x01\x00\xc5\x01\xdf\xdc\xe4\xe2\x0e\x01\x00\x10\xed\xec\x9d\xfd\x08\x01\x00\x03\xa6\xc5\xfd\xa0\x0e\x01\x009\xe0\x93\xc3\xf6\x04\x03\x009J\x01LH\xde\xd6\xe0\xbe\x0c\x02\x00\x04\x06\x11\xe7\xce\xda\xc1\x0f\x01\x00\x03\xb6\xab\xb3c\x01\x00\xd4\x01\x89\xb1\x84\xcd\x0e\x01\x00\x05\xb6\xdb\xec\x87\x05\x01\x00\x03\xff\xe6\xd5\xf2\x0b\x01\x00\x03\xbc\xc5\x89\xcf\x0b\x03\x00\xa4\x04\xbf\x05\x14\xf5\x05\x14"

I'm not sure if you can figure out something wrong with the update itself. Let me know if you need mode details.

commented

If you need the whole document change history to debug this, you will need to install y-py, ypy-websocket and jupyter-ydoc, and download this file. Then run this script to see the error:

import asyncio
import base64
import json

import y_py as Y
from ypy_websocket.ystore import SQLiteYStore
from jupyter_ydoc import YNotebook


class JupyterSQLiteYStore(SQLiteYStore):
    db_path = "jupyter_ystore.db"

ydoc = Y.YDoc()
ynotebook = YNotebook(ydoc)

async def main():
    ystore = JupyterSQLiteYStore(".notebook:Untitled.ipynb.y")
    await ystore.apply_updates(ydoc)

asyncio.run(main())
commented

This might be fixed by a version bump in #64

commented

Unfortunately, this doesn't fix the issue. You can reproduce it as described.

commented

@Horusiath Could you please have a look at this? Bumping y-crdt apparantly didn't fix the issue.

commented

I've tested this payload, on the main branch (v0.9.2) and it parsed update without issues - I cannot be 100% sure if applying it to a doc won't crash as it relies on a previous updates being processed first and I don't have y-py access atm. The decoded update looks as follows:

{
    blocks: UpdateBlocks {
        clients: {
            3352832862: [
                Item(<3352832862#9>, len: 1, origin-l: <1158732714#115>, origin-r: <1158732714#122>: deleted(1)),
                GC(<3352832862#10>-6),
                Item(<3352832862#16>, len: 1, origin-l: <3352832862#9>, origin-r: <1158732714#122>: deleted(1)),
                GC(<3352832862#17>-6),
                Item(<3352832862#23>, len: 1, origin-l: <3705480640#150>, origin-r: <3756718507#10>: <map({})>),
                Item(<3352832862#24>, len: 1, parent: <3352832862#23>, 'id' => [a584a3d2-2490-4926-b156-0e9fa9b5d718]),
                Item(<3352832862#25>, len: 1, parent: <3352832862#23>, 'cell_type' => [code]),
                Item(<3352832862#26>, len: 1, parent: <3352832862#23>, 'source' => <text>),
                Item(<3352832862#27>, len: 5, parent: <3352832862#26>: '2 + 2'),
                Item(<3352832862#32>, len: 1, parent: <3352832862#23>, 'metadata' => <map({})>),
                Item(<3352832862#33>, len: 1, parent: <3352832862#32>, 'trusted' => [true]),
                Item(<3352832862#34>, len: 1, parent: <3352832862#23>, 'execution_count' => [12]),
                Item(<3352832862#35>, len: 1, parent: <3352832862#23>, 'outputs' => <array>),
                Item(<3352832862#36>, len: 1, parent: <3352832862#35>: [{output_type: execute_result, data: {text/plain: 4}, execution_count: 12, metadata: {}}]),
            ],
        },
    },
    delete set:  {
        2602598944: [0..63),
        1955151937: [ [0..3) [23..297) ],
        2754961410: [0..94),
        353480223: [0..3),
        3705480640: [0..197),
        3965267551: [0..16),
        3827262118: [0..57),
        1322306016: [ [0..57) [74..75) [76..148) ],
        4164331367: [0..3),
        3919648905: [0..5),
        1158732714: [ [0..101) [108..178) ],
        3756718507: [ [0..206) [213..279) ],
        965947404: [0..149),
        3131908077: [ [0..7) [9..118) ],
        3051632078: [0..3),
        2410116717: [0..3),
        2009286032: [0..3),
        619221712: [0..3),
        3193271167: [0..3),
        660963412: [0..13),
        4070650870: [0..5),
        208459190: [0..212),
        1358638518: [0..3),
        1733005817: [0..31),
        891723388: [0..3),
        3118621372: [ [0..548) [703..723) [757..777) ],
        3352832862: [ [0..4) [6..23) ],
        3172524927: [0..3),
    },
}
commented

@Horusiath all the updates are provided here. Did you try that?

commented

Yeah, I've managed to recreate it, however with my python foo I'm yet in the process on getting over to test it against latest yrs version (latest y-py version on pip is 0.5.0 which uses yrs 0.7.1, while current yrs version is 0.9.2).

commented

I already checked with #64 which uses yrs 0.9.2, same error.

commented

After attaching my locally compiled version of y-py I've finally managed to trace some errors:

  1. Transaction::commit tries to squash merge_blocks found within transaction, that to clients unknown at the moment of committing. It's one of "should never happen" scenarios.
  2. It looks like some of the input updates are corrupted eg. block.id.clock = 18446744073709551615 which by no means can be considered valid.

It looks like block decoding failed for some reason filling up updates with a bad data. Another (hopefully not true) reason, could be that some pointer was dropped along the way and we're scanning invalid place in memory as if it contained block data.

I'll further investigate the issue, once I'll decode all updates within attached SQLite database file - unfortunately SQLiteYStore is basically a black box.

commented

unfortunately SQLiteYStore is basically a black box.

You can replace await ystore.apply_updates(ydoc) with the following if you want to see the updates:

    async for update in ystore.read():
        print(update)
        Y.apply_update(ydoc, update)
commented

I've released yrs v0.9.3 which should deal with this issue (problem was about trying to double merge blocks within transaction). If the issue will reappear, feel free to open it again.

commented

Thanks a lot, I've checked that it fixes the issue.

commented

@Waidhoferj @dmonad could we release ypy?