VID_HDR appears to be offset from where it belongs
iagox86 opened this issue · comments
Hey,
I'm unsure whether this is a problem with ubi_reader or with the chip I got it from, but I'm struggling to either mount or extract data from a memory dump I did.
If I just straight up try to dump it, I see this error repeated for literally every block:
guess_start_offset Found UBI magic number at 0
guess_filetype Looking for file type at 0
guess_filetype File looks like a UBI image.
UBI_File Open Path: /home/ron/image.ubi
UBI_File File Size: 237355008
UBI_File Start Offset: 0
UBI_File End Offset: 237355008
UBI_File Block Size: 135168
UBI_File read loc: 0, size: 135168
vid_hdr CRC Failed: expected 0xecfe642a got 0xffffffff
extract_blocks Block: PEB# 0: LEB# 4294967295
extract_blocks file addr: 0
extract_blocks PEB: 0 has possible issue EC_HDR [], VID_HDR [crc]
Block: PEB# 0: LEB# 4294967295
---------------------
File Offset: 0
PEB #: 0
LEB #: 4294967295
Block Size: 135168
Internal Volume: True
Is Volume Table: False
Is Valid: True
Erase Count Header
---------------------
data_offset: 4096
ec: 4264
errors: ''
hdr_crc: '0xe263b667'
image_seq: 2333686688
magic: 'UBI#'
padding: '\x00\x00\x00'
padding2: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
version: 1
vid_hdr_offset: 2048
VID Header
---------------------
compat: -1
copy_flag: 255
data_crc: 3932094975
data_pad: 4294967295
data_size: 4294967295
errors: 'crc'
hdr_crc: '0xffffffff'
lnum: 4294967295
magic: '\xff\xff\xff\xff'
padding: '\xff\xff\xff\xff'
padding2: '\xff\xff\xff\xff'
padding3: '\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
sqnum: 18446744073709551615L
used_ebs: 4294967295
version: 255
vol_id: 4294967295
vol_type: -1
I dug into the code to try and understand what headers are supposed to look like. The first block appears to be empty and have no data, but the second block does. It appears, however, that the VID header is 0x40 bytes past where ubi_reader is looking.
Here's the EB header of the second block (the block size is 0x21000):
---magic--- ver - - - --------- EC --------------
00021000: 55 42 49 23 01 00 00 00 00 00 00 00 00 00 10 0d UBI#............
vid_hdr_off data_offset -image_seq- --padding--
00021010: 00 00 08 00 00 00 10 00 8b 19 3b a0 00 00 00 00 ..........;.....
--------------------------padding--------------------------
00021020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
-----------------padding------------------- ----crc----
00021030: 00 00 00 00 00 00 00 00 00 00 00 00 c4 b3 6f 75 ..............ou
00021040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
...empty...
000217f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
If I'm reading that right, the vid_hdr_offset as 0x800, and the data_offset at 0x1000. Looks good so far!
If I fast forward to 0x800 bytes into the that block, where I'd expect to see the VID header (and where ubi_reader is reading it), I see:
00021800: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
00021810: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
00021820: 12 2e e1 ff ff ff ff ff ff ff ff ff ff ff ff ff ................
00021830: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
It appears that the checksum is set, but nothing else at all, including the 'magic'. However, if I go an additional 0x40 bytes to 0x840 from the start of the block, I see what appears to be a UBI VID header (starts with 'UBI!'):
type compat
---magic--- ver copy --vol_id--- ---lnum----
00021840: 55 42 49 21 01 01 00 00 00 00 00 00 00 00 01 18 UBI!............
----pad---- -data_size- -used_ebs-- data_padding
00021850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
data_crc padding ----------seq_num----------
00021860: 00 00 00 00 00 00 00 00 00 00 00 00 00 7e 0c ad .............~..
------------------padding------------------ ----crc----
00021870: 00 00 00 00 00 00 00 00 00 00 00 00 bc b0 9f 45 ...............E
00021880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
...empty...
00021ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
It doesn't look 100% perfect, but it does start with the "UBI!" magic and apparently has a CRC32 at the end.
If I go to 0x1000 offset from the start, it looks like it could be a data block, but I haven't gotten far enough to know what that's supposed to look like yet. I'm still working through the headers.
I haven't dug super deep into the RFC or docs just yet, just the list of headers. But I thought maybe, since you've done a lot more work with this format than myself, you might see what's going on.
Thanks for any help!
Hi iagox86,
From what you describe it sounds like maybe the manufacturer played with some numbers to get UBI out of spec. It does comes up every so often. The testing branch of ubi_reader uses an overrides.ini file, where you can specify the offset, look in the tools/ directory for an example. I made this up for a similar problem someone was having a while back and haven't gotten back to it. Your mileage my very on how well this is working.
Would it be possible to get a copy of this file? It would help in testing this feature and bringing it to master. But I understand if there is sensitive data involved.
Let me know if you have any questions on getting testing going.
-Jason
Hey, thanks for your answer! I actually worked on this all day, and talked to the folks on the mtd-tools mailing list. What was happening was out-of-band data on the flash chip. Every 0x800 bytes, the flash chip had 0x40 bytes of overhead. Once I removed it, I was able to extract the filesystem:
ubireader_extract_images -o tmp/ data.bin
I was able to mount that image and grab the files with ubifs kernel driver. So problem solved for me!
In case it helps you, though, I had trouble using ubireader_extract_files:
$ ./ubireader_extract_files -o tmp data.bin
Extracting files to: tmp/2333686688/data
read Error: LEB: 1676 is corrupted or has no data.
index Fatal: LEB: 1676, UBIFS offset: 212860440, error: Bad Read Offset Request
If I passed in -w, it would start working, with a ton of errors, but would actually extract some of the files. Unfortunately, it would also consume 100% memory then crash with SIGKILL after a minute or two:
$ ./ubireader_extract_files -o tmp data.bin -w
Extracting files to: tmp/2333686688/data
read Error: LEB: 1676 is corrupted or has no data.
index Error: Problem at file address: 210037368 extracting idx_node: unpack requires a string argument of length 20
index Error: Problem at file address: 184490008 extracting dent_node: 'utf8' codec can't decode byte 0x82 in position 2: invalid start byte
index Error: Problem at file address: 9012088 extracting idx_node: unpack requires a string argument of length 20
read Error: LEB: 317 is corrupted or has no data.
index Error: Problem at file address: 123154912 extracting idx_node: unpack requires a string argument of length 20
index Error: Problem at file address: 123158272 extracting ino_node: unpack requires a string argument of length 136
read Error: LEB: 672 is corrupted or has no data.
read Error: LEB: 762 is corrupted or has no data.
read Error: LEB: 365 is corrupted or has no data.
index Error: Problem at file address: 72175344 extracting dent_node: 'utf8' codec can't decode byte 0xc1 in position 1: invalid start byte
index Error: Problem at file address: 72176312 extracting dent_node: 'utf8' codec can't decode byte 0xfe in position 1: invalid start byte
read Error: LEB: 310 is corrupted or has no data.
index Error: Problem at file address: 72177720 extracting dent_node: 'utf8' codec can't decode byte 0xb4 in position 1: invalid start byte
read Error: LEB: 1375 is corrupted or has no data.
read Error: LEB: 1531 is corrupted or has no data.
read Error: LEB: 769 is corrupted or has no data.
read Error: LEB: 1088 is corrupted or has no data.
index Error: Problem at file address: 210034064 extracting idx_node: unpack requires a string argument of length 20
...
I really wish I could share the file with you, but it's unfortunately it's covered by NDA with my customer. But I'm happy to grab information from it if it'll help you any (I have what I need at this point, though).
Btw, it might be helpful if you check the 'magic' value of the VID block - if it's not "UBI!" it's probably bad and the CRC/etc. fields won't be valid. I doubt you could auto-detect my situation, but it'd be a helpful way to fail faster.
Good to hear you got it going. Yeah the out of band data will do that. There is a script in the testing branch's tools/ directory for removing it, if you know some info about the chip. I get excited to find something weird the manufacture did, and tend to over think things.
I think the best I could do, is add some config options and let the math factor in the OOB size to deal with it in script.
The problem here was, it is hard to tell if the vid_hdr is corrupted, offset, what ever, or the block is just empty, they look the same. I'll need to figure out a way to better deal with that. I'm guessing some numbers looked "good enough", and then it just jumped into oblivion.
Thanks for you input,
-Jason