limit cluster waste: merge LSM and LST metadata
mateuszviste opened this issue · comments
When a package is installed, the package manager needs to keep some metadata. This is stored in two files:
APPINFO/PROGRAM.LSM --> description, version, hw reqs...
PACKAGES/PROGRAM.LST --> list of files installed on disk
Idea here would be to append the list of files to the LSM file (or prepend the LST file with LSM data, whichever makes more implementation sense). This would help save some space since for many packages only a single cluster would be required instead of two.
It would also make the SvarDOS directory structure a little bit lighter.
Seems a reasonably good idea.
I would personally prefer to append LST contents to the LSM files because they are nowadays an extension name more widely known and used.
From a technical point of view it makes sense to combine those two files, but, erm, how often did you run out of clusters in the last years?
You could also concatenate all LSM files to a single file. This is called 'database' then. ;-) (sqlite?)
He is doing a great exercise in minimalism, which is one of SvarDOS concepts. Remember he reworked MORE which is now beautifully tiny.
It was a great-small achievement ;-)
From a technical point of view it makes sense to combine those two files, but, erm, how often did you run out of clusters in the last years?
Ah, the German pragmatism! :-)
It is true that this is an issue only for really, really scarce systems.
You could also concatenate all LSM files to a single file. This is called 'database' then. ;-) (sqlite?)
sqlite would be 386+, so no good, but I get the point. It is something I experimented with years ago in FDNPKG: it builds a temporary database of packages sourcing data from a set of online repositories to perform search operations etc.
The code necessary for this is big, non-trivial and potentially memory-hungry. Also, DOS is not known for having a robust filesystem and a corruption of such database would be dramatic.
I think this kind of information definitely has to stay within small, text-like files. The idea here is only to make it so every package produces 1 metadata file instead of 2. That would save... between 0.5 and 16K of disk space per package, depending on cluster size. I agree this is not a super sexy goal, but the change is also not complex, so it's a relatively easy win if I have an hour or two to spare on this.
PS. and maybe on the long run this would help keeping these pesky "package managers are bad bloatware" people at bay ;-) [humor]
He is doing a great exercise in minimalism, which is one of SvarDOS concepts. Remember he reworked MORE which is now beautifully tiny.
It was a great-small achievement ;-)
I don't deny. For its minimalism I prefer SvarDOS over FreeDOS.
From a technical point of view it makes sense to combine those two files, but, erm, how often did you run out of clusters in the last years?
Ah, the German pragmatism! :-)
Not really. Quoting myself from another bugz issue: "But I really love to throw big machines on little problems. ;-)"
But the further my energy level drops, the more my pragmatism increases.
Also, DOS is not known for having a robust filesystem and a corruption of such database would be dramatic.
I also had that in mind. I remember the (not so) funny days of Windows 9x, when your computer was hit by a corrupted registry.
I think this kind of information definitely has to stay within small, text-like files. The idea here is only to make it so every package produces 1 metadata file instead of 2. That would save... between 0.5 and 16K of disk space per package, depending on cluster size. I agree this is not a super sexy goal, but the change is also not complex, so it's a relatively easy win if I have an hour or two to spare on this.
I see.
PS. and maybe on the long run this would help keeping these pesky "package managers are bad bloatware" people at bay ;-) [humor]
Do these people really exist or are they just rumors? ;-)
pkg ver 20240131 uses a new way of keeping track of the files that belong to a package: instead of storing the list in %DOSDIR%/PACKAGES/%PKGNAME%.LST, it appends it to %DOSDIR%/APPINFO/%PKGNAME%.LSM.
This is not a "breaking" change since pkg writes in this new way, but will read from the legacy LST file if it exists. Shortly said, the migration from the old way to the new way should be completely transparent to users.
The LSM file also stores a CRC32 for every installed file. Since the format of the file was changing, it was a good occasion to add this. It will come handy for #2 .
Oh, and I've also added a new "pkg crc32" action that allows using the pkg tool to compute the CRC32 of any given file. The CRC32 code was already present since it is required for proper ZIP handling / error-checking, so I figured why not exposing it to the public. It costed about 200 bytes of binary footprint.