Try to get Gentoo to provide needed files on the distmirrors
NiKiZe opened this issue · comments
This is
gentookernelgentoo.igzoriginal initrdimage.squashfsboot.ipxethe iPXE script - once it is well tested for all platforms
This is a discussion with an already existing Bug, probably not to hard to get fix once we have good documentation here on how it could be used. https://bugs.gentoo.org/494300
From chat using https://web.libera.chat/#gentoo-releng
jpds
NiKiZe: [as a user]: I'd like that, debian has the stuff under http://ftp.uk.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/
NiKiZe
jpds yes, the current iso works as it is, so I have not found any reason to make it netboot specific? But I guess this somewhat boils down to maintenance and automation. If anyone could point me in the right direction for where minimal iso is created and copied, then maybe I could provide a patch to move forward?
iamben
look in releng.git repo, tools/catalyst-auto calls catalyst, uploading happens via the arch-specific catalyst-auto-*.conf files
im not sure if it would be best implemented in catalyst-auto, or as a separate build target in catalyst itself
NiKiZe
Thanks! Since the artifacts should already exist from building the iso I'm assuming it's best to reuse that if possible? Could you explain just a little more where I should look for "separate build target in catalyst"?
iamben
https://github.com/gentoo/catalyst/tree/master/targets
gentoo/releng
https://github.com/gentoo/releng/blob/master/tools/catalyst-auto-amd64-demeter.conf
https://wiki.gentoo.org/wiki/Catalyst
Notes for trying to use catalyst
mkdir -p /etc/portage/package.accept_keywords
echo dev-util/catalyst > /etc/portage/package.accept_keywords/catalyst
echo 'GRUB_PLATFORMS="${GRUB_PLATFORMS} emu efi-32 efi-64 pc"' >> /etc/portage/make.conf
#-boot to avoid boot partition issue with memtest
#system-bootloader needed for livecd creation
USE="-boot system-bootloader" emerge -uvN pixz catalyst genlop reptyr -j16
# python mount needed (maybe)
USE=python emerge -uvN util-linux
# grep -v ^# /etc/catalyst/catalyst.conf
# remove autoresume
# add preserve_libs
# add keepwork
# remove snapcache - first build fails with this enabled ?
options="bindist kerncache pkgcache preserve_libs keepwork seedcache"
snapshot_cache="/release/tmp/snapshots"
storedir="/release/buildroot/amd64-dev"
source_matching="loose"
# add log
port_logdir="/var/tmp/catalyst/tmp"
var_tmpfs_portage=16
# cat /etc/catalyst/catalystrc
export MAKEOPTS="-j8"
export EMERGE_DEFAULT_OPTS="--jobs 20 --load-average 9"
cd
git clone https://github.com/gentoo/releng.git
cd releng
# TODO parse /etc/catalyst/catalyst.conf to get below paths
mkdir -p /release/buildroot/amd64-dev/builds/default/
mkdir -p /release/buildroot/amd64-dev/builds/hardened/
mkdir -p /release/buildroot/amd64-dev/builds/musl
mkdir -p /release/buildroot/amd64-dev/builds/musl-hardened
mkdir -p /release/tmp/snapshots/
pushd /release/buildroot/amd64-dev/builds/default/
FILE=$(wget -q http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/ -O - | grep -o -E 'stage3-amd64-openrc-20\w*\.tar\.(bz2|xz)' | uniq)
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/$FILE
# ln -s $FILE stage3-amd64-openrc-latest.tar.xz
popd
# check expected times:
genlop -t gcc
mkdir -p /release/buildroot/amd64-dev/tmp/; \
umount /release/buildroot/amd64-dev/tmp/; \
rm -rf /release/buildroot/amd64-dev/tmp/*; mount none -t tmpfs /release/buildroot/amd64-dev/tmp/ -o size=16G,nr_inodes=1M; \
sh tools/catalyst-auto -v -v -X -l /var/lock/catalyst.lock -c tools/catalyst-auto-amd64-demeter.conf; \
umount /release/buildroot/amd64-dev/tmp/Building with snapcache enabled gives this:
*** Running command: catalyst -a -c /etc/catalyst/catalyst.conf -f /tmp/catalyst-auto.20210723T062037Z.zsSPDX/specs/stage1-openrc.spec
23 Jul 2021 08:20:44 CEST: NOTICE : Loading configuration file: /etc/catalyst/catalyst.conf
23 Jul 2021 08:20:44 CEST: NOTICE : conf_values[options] = {'preserve_libs', 'pkgcache', 'clear-autoresume', 'keepwork', 'seedcache', 'snapcache', 'bindist', 'kerncache'}
23 Jul 2021 08:20:44 CEST: NOTICE : Processing spec file: /tmp/catalyst-auto.20210723T062037Z.zsSPDX/specs/stage1-openrc.spec
23 Jul 2021 08:20:44 CEST: NOTICE : Using target: stage1
23 Jul 2021 08:20:44 CEST: NOTICE : Source file specification matching setting is: loose
23 Jul 2021 08:20:44 CEST: NOTICE : Accepted source file extensions search order: ['tar.xz', 'tpxz', 'xz', 'tar.bz2', 'bz2', 'tbz2', 'squashfs', 'sfs', 'tar.gz', 'gz', 'tar']
23 Jul 2021 08:20:44 CEST: NOTICE : Source path set to /release/buildroot/amd64-dev/builds/default/stage3-amd64-openrc-latest.tar.xz
23 Jul 2021 08:20:44 CEST: NOTICE : stage1 stage path is /release/buildroot/amd64-dev/tmp/default/stage1-amd64-openrc-20210723T062037Z/tmp/stage1root
23 Jul 2021 08:20:44 CEST: NOTICE : --- Running action sequence: unpack
23 Jul 2021 08:20:44 CEST: NOTICE : Referenced SEEDCACHE does not appear to be a directory, trying to untar...
23 Jul 2021 08:20:44 CEST: NOTICE : Clearing the chroot path ...
23 Jul 2021 08:20:44 CEST: NOTICE : Emptying directory: /release/buildroot/amd64-dev/tmp/default/stage1-amd64-openrc-20210723T062037Z
23 Jul 2021 08:20:44 CEST: NOTICE : Starting auto from /release/buildroot/amd64-dev/builds/default/stage3-amd64-openrc-latest.tar.xz
23 Jul 2021 08:20:44 CEST: NOTICE : to /release/buildroot/amd64-dev/tmp/default/stage1-amd64-openrc-20210723T062037Z (this may take some time) ..
23 Jul 2021 08:20:47 CEST: NOTICE : --- Running action sequence: unpack_snapshot
23 Jul 2021 08:20:47 CEST: NOTICE : Unpacking portage tree (this can take a long time) ...
tar: /release/tmp/snapshots/20210723T062037Z: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
23 Jul 2021 08:20:47 CEST: ERROR : Error unpacking snapshot using mode lbzip2
23 Jul 2021 08:20:47 CEST: ERROR : Exception running action sequence unpack_snapshot
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1499, in _run
getattr(self, x)()
File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 891, in unpack_snapshot
with open(snapshot_cache_hash_path, 'w') as myf:
FileNotFoundError: [Errno 2] No such file or directory: '/release/tmp/snapshots/20210723T062037Z/catalyst-hash'
23 Jul 2021 08:20:47 CEST: NOTICE : Cleaning up... Running unbind()
bootstrap stage2 fails, it was ok first time, but even after clearing all binpkgs still getting this.
disabling binpkgs might be needed if there is any pkgs left around at all
Copying stage2-chroot.sh to /tmp
copying stage2-chroot.sh to /release/buildroot/amd64-dev/tmp/default/stage2-amd64-openrc-20210724T051742Z/tmp
copying chroot-functions.sh to /release/buildroot/amd64-dev/tmp/default/stage2-amd64-openrc-20210724T051742Z/tmp
Ensure the file has the executable bit set
Running stage2-chroot.sh in chroot:
chroot /release/buildroot/amd64-dev/tmp/default/stage2-amd64-openrc-20210724T051742Z /tmp/stage2-chroot.sh
>>> Regenerating /etc/ld.so.cache...
Gentoo Linux; http://www.gentoo.org/
Copyright 1999-2020 Gentoo Authors; Distributed under the GPLv2
-------------------------------------------------------------------------------
[[ (0/3) Locating packages ]]
* Using baselayout : >=sys-apps/baselayout-2
* Using portage : sys-apps/portage
* Using os-headers : >=sys-kernel/linux-headers-5.10
* Using binutils : sys-devel/binutils
* Using gcc : sys-devel/gcc
* Using gettext : sys-devel/gettext
* Using libc : sys-libs/glibc:2.2
* Using texinfo : sys-apps/texinfo
* Using zlib : sys-libs/zlib
* Using ncurses : sys-libs/ncurses
-------------------------------------------------------------------------------
[[ (1/3) Configuring environment ]]
-------------------------------------------------------------------------------
[[ (2/3) Updating portage ]]
>>> Running pre-merge checks for sys-apps/portage-3.0.20-r6
* Checking for suitable kernel configuration options...
[ ok ]
>>> Emerging binary (1 of 1) sys-apps/portage-3.0.20-r6::gentoo
>>> Installing (1 of 1) sys-apps/portage-3.0.20-r6::gentoo
-------------------------------------------------------------------------------
[[ (3/3) Emerging packages ]]
[ebuild R ] sys-apps/baselayout-2.7
[ebuild R ] sys-libs/zlib-1.2.11-r4
[ebuild R ] sys-kernel/linux-headers-5.10
[ebuild R ] sys-devel/gettext-0.21-r1
[ebuild R ] sys-devel/binutils-2.35.2
[ebuild R ] sys-devel/gcc-10.3.0-r2
[ebuild R ] sys-libs/glibc-2.33-r1
[ebuild N ] virtual/perl-IO-1.460.0
[ebuild N ] dev-perl/Unicode-EastAsianWidth-1.330.0-r1
[ebuild N ] dev-perl/Text-Unidecode-1.300.0
[ebuild N ] dev-perl/Params-Util-1.70.0
[ebuild N ] dev-perl/Class-Inspector-1.360.0
[ebuild N ] dev-perl/File-ShareDir-Install-0.130.0
[ebuild N ] dev-perl/Exporter-Tiny-1.2.2
[ebuild N ] dev-perl/List-MoreUtils-0.428.0
[ebuild N ] dev-perl/File-ShareDir-1.116.0
[ebuild N ] dev-perl/libintl-perl-1.310.0
[ebuild R ] sys-apps/texinfo-6.7
!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:
sys-devel/gcc:10
(sys-devel/gcc-10.3.0-r2:10/10::gentoo, ebuild scheduled for merge) USE="(cxx) (multilib) nls nptl (pie) (-ada) -d -debug -doc (-fixed-point) -fortran -go -graphite (-hardened) -jit (-libssp) -lto -objc -objc++ -objc-gc -openmp -pch -pgo -sanitize -ssp -systemtap -test -vanilla -vtv -zstd" ABI_X86="(64)" pulled in by
sys-devel/gcc (Argument)
(sys-devel/gcc-10.3.0-r2-1:10/10::gentoo, installed) USE="(cxx) fortran (multilib) nls nptl openmp pch (pie) sanitize ssp (-ada) -d -debug -doc (-fixed-point) -go -graphite (-hardened) -jit (-libssp) -lto -objc -objc++ -objc-gc -pgo -systemtap -test -vanilla -vtv -zstd" ABI_X86="(64)" pulled in by
>=sys-devel/gcc-4.2:*[openmp] required by (app-crypt/libb2-0.98.1-r2-1:0/0::gentoo, installed) USE="openmp -native-cflags -static-libs" ABI_X86="(64) -32 (-x32)"
^^^^^^
>=sys-devel/gcc-4.2:*[openmp] required by (app-portage/portage-utils-0.90-1:0/0::gentoo, installed) USE="nls openmp qmanifest qtegrity -static" ABI_X86="(64)"
^^^^^^
It might be possible to solve this slot collision
by applying all of the following changes:
- sys-devel/gcc-10.3.0-r2 (Change USE: +openmp)
24 Jul 2021 07:22:22 CEST: ERROR : CatalystError: cmd(['/usr/share/catalyst/targets/stage2/stage2-controller.sh', 'run']) exited 1
ERROR:catalyst:CatalystError: cmd(['/usr/share/catalyst/targets/stage2/stage2-controller.sh', 'run']) exited 1
24 Jul 2021 07:22:22 CEST: ERROR : CatalystError: Stage build aborting due to error.
ERROR:catalyst:CatalystError: Stage build aborting due to error.
24 Jul 2021 07:22:22 CEST: ERROR : Exception running action sequence run_local
Testing if using this stage3 works around the openmp issue ...
Not working, but first test did upgrade of stage3 before using it as base + using binpkgs
FILE=$(wget -q http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/ -O - | grep -o -E 'stage3-amd64-20\w*\.tar\.(bz2|xz)' | uniq)
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/$FILE
ln -s $FILE stage3-amd64-openrc-latest.tar.xznetboot targets exists that generates multiple files.
installcd-stage2-minimal.spec could probably be made to copy out needed files as well.
The cleanest way would be to split installcd-stage2-minimal.spec up into 2 files to get a stage2 and 3
installcd-stage2.specwould build squashfs, kernel, initrd.installcd-stage3-minimal.specwould need to add bootloaders (current grub, syslinux), and generate iso.installpxe-stage3-minimal.specWould just repackage initrd+squashfs and possibly generate config files/scripts, but this still needs to generate multiple outputs?
However this probably makes to much of a disconnect between settings in the stage2 spec files and what would be needed for stage3. - not to mention all the places and documentation that would need updates in regards to livcd stage2/3.
in catalyst/targets/livecd_stage2.py function create_iso is added, which is defined in catalyst/base/stagebase.py and calls the targets/livecd-stage2/livecd-stage2-controller.sh with iso argument.
Maybe the needed copy out is best placed in targets/targets/support/create-iso.sh?
Or in the iso step of targets/livecd-stage2/livecd-stage2-controller.sh
(consider that we might want to create "gentoo.ipxe", and copy out and maybe modify both "grub.cfg" and "isolinux.cfg" (as pxelinux.cfg/default)
All this requires modification and release of catalyst.
In the end maybe the best, or at least the first solution, would be to simply extract the iso in the upload step instead?
Have tried to create PoCs to generate PXE files for minimal livecd (amd64), feedback appreciated https://bugs.gentoo.org/494300#c4
One variant modifies upload step NiKiZe/releng@b2c1ffb
Another variant modifies ISO creation NiKiZe/catalyst@33c9405
robbat2:
mattst88: proj/releng.git scripts/copy_buildsync.sh creates latest-*.txt
plus catalyst-auto in some cases
dilfridge
Chewi: /etc/catalyst/catalystrc
on demeter it's
export MAKEOPTS="-j17"
export EMERGE_DEFAULT_OPTS=" --jobs=10 --load-average=32"
export FEATURES="${FEATURES} parallel-install"
(8 core machine)