LittleFS corruption problem
arturkow2000 opened this issue · comments
Hello,
I've got problem with LittleFS corrupting itself when creating directories.
I want to create following directory structure
/
│
├── fobnail_client
│ └── dat
│ └── cert
│ └── Fobnail
│ └── 7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375
└── trussed
7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375 is a file
with 855 bytes of data. It has one xattr attached: ID 0, data 0x01 (1 byte).
At that point creating trussed directory will corrupt filesystem.
Test described below have been done on latest LittleFS release (v2.4.2,
bindings updated, trussed-dev/littlefs2-sys#4), however this
problem is present with latest binding release too. I was unable to reproduce
this issue when using raw C API.
Reproduction:
-
take lfs tool
(fromlittlefs_fixbranch for latest LittleFS version) -
build it, doesn't make difference whether it is debug or release
-
run the following commands
dd if=/dev/zero bs=4096 count=32 of=flash.bin lfs -f flash.bin --format install-certificate bad_cert.pem --trusted lfs -f flash.bin mkdir /trussed <-- thats where filesystem becomes corrupted further mount attempts will fail lfs -f flash.bin dir / -
whether corruption happens depends certificate used - it is 100% reproducible
forbad_cert.pem, so far never happened withgood_cert.pem. Both
certificates have the same size, but contents differ which affects name of
created file (hash of file contents). -
for the problem to occur certificate installation must be done in one run
which suggests corruption of in-memory state. If steps are split each into
separate commands it works:lfs -f flash.bin --format mkdir /fobnail_client lfs -f flash.bin mkdir /fobnail_client/dat lfs -f flash.bin mkdir /fobnail_client/dat/cert lfs -f flash.bin mkdir /fobnail_client/dat/cert/Fobnail lfs -f flash.bin copy-to ../bad_cert.der /fobnail_client/dat/cert/Fobnail/7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375 lfs -f flash.bin set-attr /fobnail_client/dat/cert/Fobnail/7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375 0 1 lfs -f flash.bin mkdir /trussed lfs -f flash.bin dir / -
Problem does not occur if
--trustedflag is omitted from
install-certificatecommand, even for bad certificate. Removing--trusted
flag skipsset-attrstage. Looks like the problem is triggered by writing to
a file and immediatelly setting custom attribute on that file. -
This is the dump of
flash_bin_cert_rust.bin(usingreadtree.pyscript from
littlefs repo) with bad certificate installed, and marked as trusted.littlefs v2.0 data (truncated, if it fits) gstate 0x000000000000000000000000 dir "/" mdir {0x1, 0x0} rev 2 (was 1) -> {0x1e, 0x1f} off tag type id len data (truncated) 00000008: 0ff00008 superblock 0 8 6c 69 74 74 6c 65 66 73 littlefs 00000014: 20100018 inlinestruct 0 24 00 00 02 00 00 10 00 00 ........ 0000003c: 0020040e dir 1 14 66 6f 62 6e 61 69 6c 5f fobnail_ 0000004e: 20000408 dirstruct 1 8 1e 00 00 00 1f 00 00 00 ........ 0000005a: 600ffc08 softtail . 8 1e 00 00 00 1f 00 00 00 ........ dir "/fobnail_client" mdir {0x1e, 0x1f} rev 1 (was 0) -> {0x2, 0x3} off tag type id len data (truncated) 00000014: 00200003 dir 0 3 64 61 74 dat 0000001b: 20000008 dirstruct 0 8 02 00 00 00 03 00 00 00 ........ 00000027: 600ffc08 softtail . 8 02 00 00 00 03 00 00 00 ........ dir "/fobnail_client/dat" mdir {0x2, 0x3} rev 1 (was 0) -> {0x4, 0x5} off tag type id len data (truncated) 00000014: 00200004 dir 0 4 63 65 72 74 cert 0000001c: 20000008 dirstruct 0 8 04 00 00 00 05 00 00 00 ........ 00000028: 600ffc08 softtail . 8 04 00 00 00 05 00 00 00 ........ dir "/fobnail_client/dat/cert" mdir {0x4, 0x5} rev 1 (was 0) -> {0x6, 0x7} off tag type id len data (truncated) 00000014: 00200007 dir 0 7 46 6f 62 6e 61 69 6c Fobnail 0000001f: 20000008 dirstruct 0 8 06 00 00 00 07 00 00 00 ........ 0000002b: 600ffc08 softtail . 8 06 00 00 00 07 00 00 00 ........ dir "/fobnail_client/dat/cert/Fobnail" mdir {0x6, 0x7} rev 1 (was 0) off tag type id len data (truncated) 00000014: 00100040 reg 0 64 37 34 31 35 39 38 35 62 7415985b 00000064: 20200008 ctzstruct 0 8 08 00 00 00 57 03 00 00 ....W... 00000078: 30000001 userattr 0x0 0 1 01 . -
This is what happens after creating
trusseddirectory.littlefs v2.0 data (truncated, if it fits) gstate 0x000000000000000000000000 dir "/" mdir {0x1, 0x0} rev 2 (was 1) -> {0x1f, 0x2} off tag type id len data (truncated) 00000008: 0ff00008 superblock 0 8 6c 69 74 74 6c 65 66 73 littlefs 00000014: 20100018 inlinestruct 0 24 00 00 02 00 00 10 00 00 ........ 0000003c: 0020040e dir 1 14 66 6f 62 6e 61 69 6c 5f fobnail_ 0000004e: 20000408 dirstruct 1 8 1e 00 00 00 1f 00 00 00 ........ 00000074: 00200807 dir 2 7 74 72 75 73 73 65 64 trussed 0000007f: 20000808 dirstruct 2 8 1f 00 00 00 02 00 00 00 ........ 0000008b: 600ffc08 softtail . 8 1f 00 00 00 02 00 00 00 ........ dir "/trussed" mdir {0x1f, 0x2} rev 2 (was 1) -> {0x1e, 0x1f} off tag type id len data (truncated) 00000008: 600ffc08 softtail . 8 1e 00 00 00 1f 00 00 00 ........ dir "/fobnail_client" mdir {0x1f, 0x1e} rev 2 (was 1) -> {0x1e, 0x1f} off tag type id len data (truncated) 00000008: 600ffc08 softtail . 8 1e 00 00 00 1f 00 00 00 ........ *** cycle detected {0x1f, 0x1e}! *** -
This problem occurs only when marking certificate as trusted and this is the
difference beetween image with trusted certificate (corrupted) and untrusted
certificate--- hex.bad_untrusted 2022-04-05 15:11:02.089691364 +0200 +++ hex.bad 2022-04-05 15:10:52.585639558 +0200 @@ -1541,8 +1541,8 @@ 00006040 63 62 64 34 35 36 36 66 34 66 31 38 66 33 39 31 |cbd4566f4f18f391| 00006050 31 33 37 35 20 00 00 40 70 1f fc 04 70 aa e6 c3 |1375 ..@p...p...| 00006060 70 2f fc 0c 08 00 00 00 57 03 00 00 70 2f fc 0c |p/......W...p/..| -00006070 a6 5b aa 48 ff ff ff ff ff ff ff ff ff ff ff ff |.[.H............| -00006080 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| +00006070 a6 5b aa 48 60 0f fc 05 01 60 0f fc 06 4d 2b d6 |.[.H`....`...M+.| +00006080 33 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |3...............| 00006090 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 000060a0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 000060b0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
-
same thing from
readtree.pyperspective--- untrusted 2022-04-05 15:26:12.690029226 +0200 +++ trusted 2022-04-05 15:26:19.502059088 +0200 @@ -76,10 +76,12 @@ 0000005c: 500ffc04 crc 0x0 . 4 70 aa e6 c3 p... 00000064: 20200008 ctzstruct 0 8 08 00 00 00 57 03 00 00 ....W... 00000070: 500ffc04 crc 0x0 . 4 a6 5b aa 48 .[.H - 00000078: aff003fb struct 0xff 0 1019 ff ff ff ff ff ff ff ff ........ - 00000477: 500ffc04 crc (bad) . 4 ff ff ff ff .... - 0000047f: aff003fb struct 0xff 0 1019 ff ff ff ff ff ff ff ff ........ - 0000087e: 500ffc04 crc (bad) . 4 ff ff ff ff .... - 00000886: aff003fb struct 0xff 0 1019 ff ff ff ff ff ff ff ff ........ - 00000c85: 500ffc04 crc (bad) . 4 ff ff ff ff .... - 00000c8d: aff003fb struct 0xff 0 1019 ff ff ff ff ff ff ff ff ........ + 00000078: 30000001 userattr 0x0 0 1 01 . + 0000007d: 500ffc07 crc 0x0 . 7 4d 2b d6 33 ff ff ff M+.3... + 00000088: aff003f8 struct 0xff 0 1016 ff ff ff ff ff ff ff ff ........ + 00000484: 500ffc07 crc (bad) . 7 ff ff ff ff ff ff ff ....... + 0000048f: aff003f8 struct 0xff 0 1016 ff ff ff ff ff ff ff ff ........ + 0000088b: 500ffc07 crc (bad) . 7 ff ff ff ff ff ff ff ....... + 00000896: aff003f8 struct 0xff 0 1016 ff ff ff ff ff ff ff ff ........ + 00000c92: 500ffc07 crc (bad) . 7 ff ff ff ff ff ff ff ....... + 00000c9d: aff003f8 struct 0xff 0 1016 ff ff ff ff ff ff ff ff ........
Looks like presence of user attribute causes mkdir to corrupt filesystem, what
puzzles me more is the connection with file contents (corruption doesn't happen
with "good" certificate). Also there are some differences in behaviour between C
and Rust - Rust code starts block allocation from the last 2 blocks (except root
directory which is always at {1, 0}), fobnail_client directory is allocated at
{30, 31} in case of Rust and {10, 11} in case of C. Shouldn't LittleFS allocate
the same block?
I upload here both good and bad certificates as well as various images: with
trusted and untrusted bad certificate, example of corrupted image and C code
(which does the same but without corrupting fs).
lfs_c.zip
images.zip