`pkg_comp auto some/package` overwrites pkg_summary
truist opened this issue · comments
Building individual packages via pkg_comp seems to end up overwriting the pkg_summary file(s) with just the packages that were built during that session, even if lots of other packages were already present in the package directory.
So if you e.g. just want to add a package to your system, and you build it via e.g. pkg_comp auto -f some/package, and follow that with pkgin update, you end up with a very short list of available packages.
This is because of how pbulk works: you have to list all packages in the limited list file (list.txt) all the time so that they end up in the pkg_summary. In other words: if you just want to build an extra package, you need to tell pbulk to build everything plus that one package.
pkg_comp build already handles this by taking the existing list.txt in the sandbox and appending the newly-requested packages to it. We'd do the same for pkg_comp auto by ensuring that the limited list generated when you do pkg_comp auto pkg1 ends up as being the union of pkg1 plus the value of AUTO_PACKAGES. In the general case, this is fast because all previously-built packages are taken from binaries and thus pbulk doesn't do anything for them.
But I'm starting to think that this approach is wrong. First because it's confusing: if you are requesting to build only a package, you shouldn't see builds for anything else. And second because it risks being very slow if you happen to have updated pkgsrc and such update actually triggers rebuilds of unrelated packages.
I don't see a way to make pbulk account for existing packages when generating pkg_summary other than by messing with the internal contents of the success file, and doing that seems fragile.
Maybe pkg_comp should generate pkg_summary files itself after pbulk has completed so that it can account for all existing binary packages?
I didn't realize that it was appending to list.txt; that is somewhat surprising to me.
Isn't pkg_summary basically just a "stream" of the packages? (i.e. there's no header?) If so, could you just append the new data to the end of the existing pkg_summary, at the end of the build?
Or yes, I expect you could just regenerate the file from the packages directory. I've done that myself (by hand) a few times, to work around this, and it seems fine.
As I understand it, the problem is that pbulk regenerates the pkg_summary file from scratch based on the packages built via list.txt. That's why pkg_comp goes through some contortions to ensure list.txt contains "everything" (so that pkg_summary is complete at the end). This was working for build but not for auto.
Then I found a comment in pkg_comp itself saying that pbulk wipes binary packages it was not asked to build. This doesn't seem to match what you observed, so I need to verify what really is going on.
Making pkg_comp generate pkg_summary on its own after a pbulk build seems the simplest approach though. I'm looking into it.
OK, I think this should be fixed now. Ended up implementing custom generation of pkg_summary files.
Looks great - thank you! I should have opportunity to test it in the next few days; I'll let you know if I run into anything unexpected. I reviewed the commit and it looks great.