This is a collection of hacky scripts to mass-build packages with some variation set-up in the build environment.
The specific variations here change frequently, but the scripts are pretty resilient, mostly because there's nearly nothing here: There's about 25 lines of Makefile, with only a few weird makeisms, and a couple of Dockerfile statements. That's it.
First, we find some packages. I do this by grepping all-build-deps-amd64.txt, which is generated by dose, and a whole lot of terrible Makefiles/Dockerfiles.
Then, we call make with all the packages we want, with .pkg rammed on the end:
% P=$(<all-build-deps-amd64.txt fgrep default-jdk | cut -d' ' -f 1 | sed 's/$/.pkg/')
make -j8 ${=P}
make goes off and does all the builds, leaving thousands of megabytes of docker
container lying around (maybe it should run it with --rm? Ho ho.), and some logs,
named:
foo.pkgforfoosucceedingfoo.pkg.failiffoofailed.foo.pkg.wipiffoois still building (or if the machine locked up during a build).
Re-running make will rebuild everything that failed the first time.
I've tried to make make make noise about what it's doing, but there's a problem somewhere:
sometimes it will hang silently during one of the container builds. Shrug, don't care.
One of the patches kicking around in this repo is to hack the JDK to accept older
source levels, and silently upgrade them. It's supposed to print to stderr, but I
don't think this works in many cases. Also, it doesn't catch the <javadoc source=
case that the ant hacks are currently missing.
Also in this repo is a script named classify.sh, which tries to guess the problem by
grepping the logs. It's awful, ignore it. It needs:
bzcat reproducible.json.bz2 | jq -r '.[]|select(.architecture=="amd64" and .status != "reproducible" and .status != "unreproducible" and .suite=="unstable")|.package' > .jenkins
diff -u <(ls -1 *.fail | sed 's/.pkg.fail$//') .jenkins | egrep '^-' | cut -c2- > .broken
As it's the future, we can rent big machines and use them to
build. Building most of the 1,500 default-jdk deps takes about
two hours on an i3.8xlarge, which costs about $3 on the
"defined duration" spot market.
I use the Ubuntu Xenial images, 'cos we only really care about Docker and a modern kernel for it. Sorry, Debian!
The notes below are a simplified version of my WIP ec2 tuning document.
Get us a whole load of discs:
#sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 /dev/xvdb /dev/xvdc && \
sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=4 /dev/nvme?n1 && \
sudo mkfs.xfs -K /dev/md0 && \
sudo mkdir /mnt/data && \
sudo mount -o nobarrier /dev/md0 /mnt/data
Install Docker from upstream as the missing flags in the old versions drive me up the wall:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
sudo apt update && \
sudo apt install -y docker-ce git wget equivs tmux && \
sudo systemctl stop docker
Move the docker data onto our monstrous partition:
sudo mv /var/lib/docker /mnt/data/ &&
sudo ln -s /mnt/data/docker /var/lib
Grab this repo and the dependency list:
git clone https://github.com/FauxFaux/debjdk9 && \
cd debjdk9 && \
wget https://b.goeswhere.com/default-jdk-dependencies-2017-08-22.lst
Can speed up this stage by copying any built debs over,
so you don't have to wait for them to build on your expensive machine.
And, in tmux, cripple the proxy (as deb.debian.org is fine), and go!
tmux
:>base/apt.conf && \
make -j $(nproc) $(cat default-jdk-dependencies-*)