Add make target to download sources
voltagex opened this issue · comments
Having a make target that only downloads (and probably unpacks) sources would make caching builds in CI easier.
This is probably possible with the current Makefile but I am not familiar enough with makefile syntax
Also looking for this!
Same here though with Makefile familiarity...
Easiest way to get everything you need is to run make extract_all TARGET=... ; make clean and tar up the sources/ directory, then untar during your build pipeline.
The file extraction is slow and noisy but will grab everything - you don't actually need to do this though, the sources can be pulled directly given a filename under sources/. If you use environment variables for versions, you can reference them in your config.mak or via make arguments.
musl-cross-make # musl-cross-make $ make sources/gcc-9.4.0.tar.xz TARGET=x86_64-linux-musl
mkdir -p sources
mkdir -p sources/gcc-9.4.0.tar.xz.tmp
cd sources/gcc-9.4.0.tar.xz.tmp && wget -c -O gcc-9.4.0.tar.xz https://ftpmirror.gnu.org/gnu/gcc/gcc-9.4.0/gcc-9.4.0.tar.xz
Connecting to ftpmirror.gnu.org (209.51.188.200:443)
Connecting to mirror.us-midwest-1.nexcess.net (208.69.120.125:443)
saving to 'gcc-9.4.0.tar.xz'
gcc-9.4.0.tar.xz 100% |*******************************************************************************************************************************************************************| 69.0M 0:00:00 ETA
'gcc-9.4.0.tar.xz' saved
cd sources/gcc-9.4.0.tar.xz.tmp && touch gcc-9.4.0.tar.xz
cd sources/gcc-9.4.0.tar.xz.tmp && sha1sum -c /usr/local/crosware/tmp/musl-cross-make/hashes/gcc-9.4.0.tar.xz.sha1
gcc-9.4.0.tar.xz: OK
mv sources/gcc-9.4.0.tar.xz.tmp/gcc-9.4.0.tar.xz sources/gcc-9.4.0.tar.xz
rm -rf sources/gcc-9.4.0.tar.xz.tmp
If you're doing CI, use environment variables; you generate a config.mak in that case as well; e.g., set your target, GCC version, etc. via the environment:
musl-cross-make # ( export GCC_VER=11.2.0 TARGET=x86_64-linux-musl ; make sources/gcc-${GCC_VER}.tar.xz )
This will pull an individual file, checksum, and drop it in the sources/ directory.
extract_all output:
musl-cross-make # test -e sources && find sources/ -type f ; make extract_all TARGET=x86_64-linux-musl &>/dev/null ; make clean ; ls -lA sources/
rm -rf gcc-* binutils-* musl-* gmp-* mpc-* mpfr-* isl-* build build-* linux-*
total 98492
-rw-r--r-- 1 root root 21490848 Apr 19 01:28 binutils-2.33.1.tar.xz
-rw-r--r-- 1 root root 36159 Apr 19 01:28 config.sub
-rw-r--r-- 1 root root 72411232 Apr 19 01:27 gcc-9.4.0.tar.xz
-rw-r--r-- 1 root root 2386766 Apr 19 01:28 gmp-6.1.2.tar.bz2
-rw-r--r-- 1 root root 1078912 Apr 19 01:28 linux-headers-4.19.88-1.tar.xz
-rw-r--r-- 1 root root 701263 Apr 19 01:28 mpc-1.1.0.tar.gz
-rw-r--r-- 1 root root 1652074 Apr 19 01:28 mpfr-4.0.2.tar.bz2
-rw-r--r-- 1 root root 1080786 Apr 19 01:28 musl-1.2.5.tar.gz
musl-cross-make # git status --ignored
On branch master
Your branch is up to date with 'origin/master'.
Ignored files:
(use "git add -f <file>..." to include in what will be committed)
sources/
nothing to commit, working tree clean
sources/ should survive a make clean but will deleted with a make distclean or git clean -fdx
Thanks for the thorough writeout @ryanwoodsmall, much appreciated!
I went with your suggestion to spell out each package version as environment variables, then running make sources/...tar.... for each. Works great!