cannot decompress big-endian filesystem on little-endian machine
mirabilos opened this issue · comments
Older versions yielded:
Filesystem on /tmp/filesystem.image is (4:0), which is a later filesystem version than I support!
Current git HEAD yields:
Reading a different endian SQUASHFS filesystem on /tmp/filesystem.image
FATAL ERROR: Can't find a valid SQUASHFS superblock on /tmp/filesystem.image
@mirabilos How odd. Having just looked at the code recently, I'm surprised that you have something that claims to be 4.0 format from that first error message... (I thought 4.0 was always little-endian on disk) And yet triggers the "reading a different endian" unsquashfs-3 code path, on your little-endian machine...?!
Can you provide instructions on how you created filesystem.image? Or put the filesystem image online somewhere for others to look at?
If it is sensitive information, and not readily reproducible, even putting up the header (first hundred bytes or so) would be illuminating.
$ dd if=test.squashfs4 bs=96 count=1 | hexdump -C
1+0 records in
1+0 records out
96 bytes copied, 3.9167e-05 s, 2.5 MB/s
00000000 68 73 71 73 c7 00 00 00 54 f8 fb 64 00 00 02 00 |hsqs....T..d....|
00000010 0d 00 00 00 01 00 11 00 c0 00 01 00 04 00 00 00 |................|
00000020 54 19 00 00 00 00 00 00 e1 65 3b 00 00 00 00 00 |T........e;.....|
00000030 d9 65 3b 00 00 00 00 00 ff ff ff ff ff ff ff ff |.e;.............|
00000040 de 54 3b 00 00 00 00 00 8a 5c 3b 00 00 00 00 00 |.T;......\;.....|
00000050 4f 64 3b 00 00 00 00 00 cb 65 3b 00 00 00 00 00 |Od;......e;.....|
00000060
I actually untarred a Fritzbox OS upgrade image from AVM, being curious about what’s in them. I did manage to extract it in the end, with sasquatch-v4be, but to even get there was an undocumented PITA…
@mirabilos
Ah, the fritzbox folks have a patched, customized bigendian variant of squashfs 4.0.
Many folks have created tools to work with that variant. See Freetz and Freetz-NG
Google for unsquashfs4-avm-be (example 1 and 2 ) to see others doing this.
You can see the patches for squashfs4-be from freetz-ng here
Below is a dockerized, instant gratification script for looking at the fritzbox-7590 root-on-squashfs filesystem. After it runs, you'll have an unsquashed squashfs-root.tgz tarball. Adjust the URL for other fritzbox models. This script also exports the unsquashfs4-avm-be utility after it's built for you to play with.
#!/bin/bash
URL=https://download.avm.de/fritzbox/fritzbox-7590/other/fritz.os/FRITZ.Box_7590-07.57.image
DOCKER_BUILDKIT=1 docker build --target=output --output=. - <<EOF
FROM ubuntu:22.04 as build
RUN apt update && \
TZ=Etc/UTC DEBIAN_FRONTEND=noninteractive \
apt install -y sudo git tzdata
RUN useradd -ms /bin/bash build
WORKDIR /home/build
RUN sudo -u build git clone https://github.com/Freetz-NG/freetz-ng.git
WORKDIR /home/build/freetz-ng
RUN tools/prerequisites install -y
RUN sudo -u build make tools
RUN curl ${URL} | tar -x
RUN ./tools/unsquashfs4-avm-be var/tmp/filesystem.image
RUN tar cvzf squashfs-root.tgz -C squashfs-root .
FROM scratch as output
COPY --from=build /home/build/freetz-ng/tools/unsquashfs4-avm-be /
COPY --from=build /home/build/freetz-ng/squashfs-root.tgz /
EOF
@plougher should probably close this issue - it is asking about one of the non-upstreamed descendants of squashfs.
I didn't realize the squashfs family was still fragmented. v4.0 came out in 2009(?) and standardized on pure little-endian for the disk format. I guess folks like Fritzbox on a bigendian MIPS carried forward a bigendian variant of 4.0 via patches?
Is there that much of a performance hit for the bigendian mips users to use the vanilla format? (just a few byteswaps in the header fields, right...)
If there's a case for it to exist, would @plougher consider upstreaming support for a bigendian squashfs 4.0?
Or perhaps lack of interoperability w/ non-patched squashfs-tools was considered a security-through-obscurity feature by the fritzbox team?
In any case, with the dominance of little-endian across x86_64, aarch64, and risc-v - we only see some niche bigendian headaches on mips, s390x, etc.
The Squashfs 4.0 format has fragmented much less than previous formats because I designed it to accommodate future enhancements without needing changes (i.e. different compression algorithms, extended attributes, etc.).
I perceive people making changes to the 4.0 format (such as producing a big-endian version) is less due to performance and more due to the desire to produce a private variant which cannot be read with my upstream kernel code and utilities (and other peoples' implementations). But they never tell me (the author) why they did it, and so I cannot speak for them.
As to whether I should officially recognise these un-official variants, the obvious answer is they are perhaps deliberately un-official, and making them official will encourage them to make further changes to make their variant again un-official. This is a situation where it is best to ignore it and do nothing, because doing anything else will make things worse.