Giters

sripathikrishnan / redis-rdb-tools

Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON

Home Page:https://rdbtools.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unknown opcode in custom rdbs

posix4e opened this issue · comments

commented

It seems as though no matter what keys I filter for, since I loaded a redis with a custom rdb there's no way to analyze this file with your parser. I'm curious if you could provide a mode where it's automatically skipped like rdb --command protocolskiperrors

Example of failed command rdb --command protocol -k 'anything' customrdb > /dev/null

Traceback (most recent call last):
  File "/home/posix4e/.local/bin/rdb", line 11, in <module>
    sys.exit(main())
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/cli/rdb.py", line 101, in main
    parser.parse(dump_file)
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 394, in parse
    self.parse_fd(open(filename, "rb"))
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 463, in parse_fd
    self.skip_object(f, data_type)
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 656, in skip_object
    self.skip_module(f)
  File "/home/posix4e/.local/lib/python3.7/site-packages/rdbtools/parser.py", line 834, in skip_module
    raise Exception("Unknown module opcode %s" % opcode)
Exception: Unknown module opcode 8198801549535663106
commented

This error indicates that it's unable to parse the RDB file, so it can't skip this and keep reading the file.

Please describe how you created this RDB, which version of redis, and which module. and if possible upload an example RDB that reproduces this.

commented

I believe the issue is our custom rdb. Once I delete the custom rdb keys it goes away. We used https://github.com/kristoff-it/redis-cuckoofilter+

commented

@posix4e what do you mean by custom rdb? you mean a module data type (using redis module API)? these should be encoded in a way that they can be skipped.
which version of redis do you use and which APIs?

commented
commented

@kristoff-it any ideas why your crazy module causes that? want to help us get to the bottom of this?

commented

IIRC i tested redis-rdb-tools with rejson keys and it worked.
i assume that rdb isn't corrupted and can be read back by redis.
can redis-check-rdb parse this rdb file?
can you maybe post a very small rdb that reproduce it?

commented

I just tested on my computer loading the latest released version of my module.
I created a couple of keys, added stuff to them, called SAVE and then rebooted the server.
All worked fine, keys got loaded back in and the content seemed to be properly preserved.

Here is where I make use of the custom type API: https://github.com/kristoff-it/redis-cuckoofilter/blob/4c617941c529d8f9a59bf4562883c792607eaef8/src/t_cuckoofilter.zig#L119-L128

I don't really do anything non-standard AFAIK. I implement 3 different data types (one for each fingerprint size the module supports), and that's pretty much it.

I also tried to parse my rdb file with rdbtools (freshly installed from pip) and I got no errors.

commented
commented

@posix4e this is what I did before replying yesterday. rdb-tools doesn't need to know about my module specifically to be able to parse the RDB file.

Can you please make sure you're using the latest release of redis-cuckoofilter (v1.1.0) and try to reproduce the problem in a RDB file to share with us, so that @oranagra can take a look?

I haven't been able to reproduce it on my own.

commented