Some unsquashed files have different size than original
duaneear opened this issue · comments
I am finding differences between some files in a .squashfs image and the original directory (a mounted, offline, Windows Primary partition. Whether I mount the squashfs image or unsquash it, comparison to the original directory shows some files are different. So far it appears the only differences are changes to the size and permissions of binary files. I have yet to examine the contents of these files but "ls -l" shows some of the differing files to have identical parameters except for their size and the "others" write permission has been disabled.
I am using "mksquashfs version 4.5 (2021/07/22)" on a KDE neon 5.26 distribution and squashfs-tools-ng from
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libsquashfs1 amd64 1.1.3-2 [40.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 squashfs-tools-ng amd64 1.1.3-2 [84.4 kB]
To assess this, you need to answer some questions, and do some more testing ...
-
Are the affected files the same when you mount or Unsquashfs the filesystem? If they're different that will imply a issue with reading the filesystem. If they're the same it will imply the issue is with creating the filesystem.
-
If they're the same, the next stage is to determine if the issue is with reading the source filesystem (mounted, offline, Windows Primary partition), or the actual program itself.
-
Try creating the Squashfs filesystem with both Mksquashfs and Gensquashfs. Are the affected files the same when mounting or unsquashing. If they are, the issue is most likely to be with reading the files from the source filesystem. If they're different, the issue is most likely with the program (either Mksquashfs or Gensquashfs).
I cannot proceed without getting answers to the above questions.
A third test would be for you to use TAR to create an archive of the filesystem, and see if you get the same or different results than Squashfs.
Sorry, I replied to the emails only. Here are my responses:
-
Yes, the squashfs file generated by mksquashfs, mounted and unsquashed, produces the same erroneous file lengths for the five files I checked (I don't need a full directory comparison, do I?)
-
The squashfs file generated by gensquashfs, mounted, appears to produce the correct file length. I will run a full directory comparison tomorrow.
Using diff to compare the mounted, gensquashfs-generated squashfs file, the previous errors in the mounted, mksquashfs-generated file are absent. However, some Microsoft log files could not be produced by the squashfs mounter (is that the best way to name that?). If I tried to open these files with an editor, the editor would hang; similar log files could be opened by the editor and I didn't have this problem with the mounted, mksquashfs-generated file. These problematic log files are under various Microsoft identity DBStore/Logfiles sudirectories and named edbres00001.jrs and edbres00002.jrs. If needed, I suppose I could post the original ~6MB files to Google Drive. In order to compare the directories, I excluded all files with these names.
diff has complained that five files under various CryptnetUrlCache directories are fifos and would not compare them.
I have not yet attempted to unsquash the gensquashfs-generated squashfs file.
I may explicitly compare the binary files that had different lengths in the mounted, mksquashfs-generated file; I think diff's directory comparison is sufficient to show mksquashfs did not properly compress any of those files you would like posted as well.
To question one, I have diffed the five affected files I reported to have the same erroneous lengths, those in the mounted versus those in the Unsquashed Mksquashfs-generated filesystem. They are the same.
To question two, I have diffed those same affected files but in the original filesystem versus in the mounted Gensquashfs-generated filesystem. They are the same.
However, as last noted, there are other problems in the mounted Gensquashfs-generated filesystem (I just noticed that the Gensquashfs-generated filesystem is ~20% smaller than the Mksquashfs-generated filesystem). Perhaps these problems arise from incompatibility of Gensquashfs and the squashfs mounter.
I have used TAR to create an archive of the filesystem and used tar --compare to compare the original and the archive. They are the same.
Please let me know what more you need.
OK, your results are what I expected to happen.
They show that both Mksquashfs and Gensquashfs don't work correctly. But the common factor here is your source filesystem and your files. That suggests that there is something unexpected happening reading your source filesystem and your files that both Mksquashfs and Gensquashfs can't deal with. So what is wrong with either your source filesystem or your files to cause that to happen?
You say it is a "a mounted, offline, Windows Primary partition", which doesn't tell me a lot to be honest, except it is a non-native Linux filesystem with files that have not been created by a Linux system. This suggests there's something there that a Linux application can't handle, hence the failures.
Now the issue is, is it the non-Linux filesystem causing the issues, or the non-Linux files creating the issues. If you're thinking what is the difference, think about this. If you put the same files both on a non-Linux filesystem and on a Linux filesystem does anything different happen? If the same files on a Linux filesystem read correctly, it is the non-Linux filesystem which is the problem. If the same files on both a non-Linux and Linux filesystem cause issues it is the files themselves which is causing the issue.
So to get a handle on what is happening, it is useful to determine which of the above is happening.
The perhaps easiest way is to copy your files on to a Linux filesystem and run Mksquashfs and Gensquashfs on that. Do they now create a correct Squashfs filesystem or not? Obviously you have to ensure the Linux filesystem copy is entirely correct, because even the copying onto a Linux filesystem may fail in the same way if you're using Linux utilities to do the copying.
One interesting additional fact which you mention is that TAR archives are correct. Which means the TAR application can deal correctly with the non-Linux filesystem and/or the non-Linux files.
Very importantly, did you generate the TAR archive on Linux from the filesystem mounted on Linux, or did you generate the TAR archive on Windows. The difference is whether you were using the Linux filesystem reading code.
Additionally importantly, Mksquashfs version 4.5 or later (run mksquashfs -version to tell), can create Squashfs filesystems directly from an uncompressed tar archive.
So you should be able to create a Squashfs filesystem from your uncompressed TAR archive as follows
% cat "your tar archive" | mksquashfs - output.sqsh -tar
If your TAR archive is zlib (gzipped) compressed, you can uncompress it and pass it to Mksquashfs as follows
% zcat "your tar archive" | mksquashfs - output.sqsh -tar
The two facts are:
- If that works then it is a workaround that creates a correct Squashfs filesystem
- It will indicate that it is the source filesystem that is the problem.
It if it doesn't work, then it is the files themselves which is the problem.
Either way the results will be interesting.
One question I forgot to ask.
What is the non-Linux filesystem, and how are you mounting it and reading it on Linux? In otherwords what filesystem code is being used to read it.
Non-Linux filesystems should behave no differently to native Linux filesystems when read by a user-space application. If there is a difference it suggests something more fundamental is going on here.
This is why seeing if the problem still exists when reading from a native Linux filesystem or from an intermediately generated TAR file is so important. It either rules in or rules out reading from the non-Linux filesystem as the problem.
Closed due to lack of response. I assume the issue has either gone away, the TAR archive workaround works, or the person who raised the issue doesn't want to spend anymore time finding the root cause.