Docker build failing with temporary file not found
openoms opened this issue · comments
Using the config: https://github.com/mkaczanowski/packer-builder-arm/blob/master/boards/raspberry-pi/raspbian.json with the docker image from master and same with the latest v1.0.4, I cannot get past this error both locally and in github actions:
docker run --rm --privileged -v /dev:/dev -v ${PWD}:/build \
mkaczanowski/packer-builder-arm:master@sha256:1c5b6a23bf7e5a02413bd2bb094422ce6bb3ea94f6873a70aacdad25c767df54 \
build raspbian.json
uname -a: Linux bd31978f800b 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 x86_64 x86_64 GNU/Linux
{
"supported": [
"linux/amd64",
"linux/arm64",
"linux/riscv64",
"linux/ppc64le",
"linux/s390x",
"linux/386",
"linux/mips64le",
"linux/mips64",
"linux/arm/v7",
"linux/arm/v6"
],
"emulators": [
"qemu-aarch64",
"qemu-arm",
"qemu-mips64",
"qemu-mips64el",
"qemu-ppc64le",
"qemu-riscv64",
"qemu-s390x"
]
}
running /bin/packer build arm64-rpi.json
2022/11/26 09:06:26 [INFO] Packer version: 1.8.4 [go1.18.5 linux amd64]
2022/11/26 09:06:26 [TRACE] discovering plugins in /usr/bin
2022/11/26 09:06:26 [DEBUG] Discovered plugin: arm = /usr/bin/packer-builder-arm
2022/11/26 09:06:26 [INFO] using external builders: [arm]
2022/11/26 09:06:26 [TRACE] discovering plugins in /root/.config/packer/plugins
2022/11/26 09:06:26 [TRACE] discovering plugins in .
2022/11/26 09:06:26 [TRACE] discovering plugins in /build/.packer_plugins
2022/11/26 09:06:26 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/11/26 09:06:26 [INFO] PACKER_CONFIG env var set; attempting to open config file: /root/.packerconfig
2022/11/26 09:06:26 [WARN] Config file doesn't exist: /root/.packerconfig
2022/11/26 09:06:26 [INFO] Setting cache directory: /build/.packer_cache
e: Running in background, not using a TTY
2022/11/26 09:06:26 [TRACE] Starting external plugin /usr/bin/packer-builder-arm
2022/11/26 09:06:26 Starting plugin: /usr/bin/packer-builder-arm []string{"/usr/bin/packer-builder-arm"}
2022/11/26 09:06:26 Waiting for RPC address for: /usr/bin/packer-builder-arm
2022/11/26 09:06:27 packer-builder-arm plugin: 2022/11/26 09:06:27 Plugin address: unix /tmp/packer-plugin2156715463
2022/11/26 09:06:27 packer-builder-arm plugin: 2022/11/26 09:06:27 Waiting for connection...
2022/11/26 09:06:27 Received unix RPC address for /usr/bin/packer-builder-arm: addr is /tmp/packer-plugin2156715463
2022/11/26 09:06:27 packer-builder-arm plugin: 2022/11/26 09:06:27 Serving a plugin connection...
2022/11/26 09:06:27 [TRACE] Starting internal plugin packer-provisioner-shell
2022/11/26 09:06:27 Starting plugin: /usr/bin/packer []string{"/usr/bin/packer", "plugin", "packer-provisioner-shell"}
2022/11/26 09:06:27 Waiting for RPC address for: /usr/bin/packer
2022/11/26 09:06:29 packer-provisioner-shell plugin: [INFO] Packer version: 1.8.4 [go1.18.5 linux amd64]
2022/11/26 09:06:29 packer-provisioner-shell plugin: [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/11/26 09:06:29 packer-provisioner-shell plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: /root/.packerconfig
2022/11/26 09:06:29 packer-provisioner-shell plugin: [WARN] Config file doesn't exist: /root/.packerconfig
2022/11/26 09:06:29 packer-provisioner-shell plugin: [INFO] Setting cache directory: /build/.packer_cache
2022/11/26 09:06:29 packer-provisioner-shell plugin: args: []string{"packer-provisioner-shell"}
2022/11/26 09:06:29 packer-provisioner-shell plugin: Plugin address: unix /tmp/packer-plugin2426615454
2022/11/26 09:06:29 packer-provisioner-shell plugin: Waiting for connection...
2022/11/26 09:06:29 Received unix RPC address for /usr/bin/packer: addr is /tmp/packer-plugin2426615454
2022/11/26 09:06:29 packer-provisioner-shell plugin: Serving a plugin connection...
2022/11/26 09:06:29 Preparing build: arm
2022/11/26 09:06:30 Build debug mode: false
2022/11/26 09:06:30 Force build: false
2022/11/26 09:06:30 On error:
2022/11/26 09:06:30 Waiting on builds to complete...
2022/11/26 09:06:30 Starting build run: arm
2022/11/26 09:06:30 Running builder: arm
2022/11/26 09:06:30 [INFO] (telemetry) Starting builder arm
arm: output will be in this color.
==> arm: Retrieving rootfs_archive
==> arm: Trying https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false
2022/11/26 09:06:30 packer-builder-arm plugin: 2022/11/26 09:06:30 Acquiring lock for: https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspbian%2Fimages%2Fraspbian-2019-09-30%2F2019-09-26-raspbian-buster.zip.sha256 (/build/.packer_cache/3ca77ef991b525741eaf282429f3466d549981ed.zip.lock)
==> arm: Trying https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspbian%2Fimages%2Fraspbian-2019-09-30%2F2019-09-26-raspbian-buster.zip.sha256
2022/11/26 09:06:36 packer-builder-arm plugin: 2022/11/26 09:06:36 Leaving retrieve loop for rootfs_archive
==> arm: https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspbian%2Fimages%2Fraspbian-2019-09-30%2F2019-09-26-raspbian-buster.zip.sha256 => /build/.packer_cache/3ca77ef991b525741eaf282429f3466d549981ed.zip
arm: unpacking /build/.packer_cache/3ca77ef991b525741eaf282429f3466d549981ed.zip to raspberry-pi.img
arm: mapping image raspberry-pi.img to free loopback device
arm: image raspberry-pi.img mapped to /dev/loop16
arm: mounting /dev/loop16p2 to /tmp/68291840
arm: mounting /dev/loop16p1 to /tmp/68291840/boot
arm: running extra setup
arm: mounting /dev with: [mount --bind /dev /tmp/68291840/dev]
arm: mounting /devpts with: [mount -t devpts /devpts /tmp/68291840/dev/pts]
arm: mounting proc with: [mount -t proc proc /tmp/68291840/proc]
arm: mounting binfmt_misc with: [mount -t binfmt_misc binfmt_misc /tmp/68291840/proc/sys/fs/binfmt_misc]
arm: mounting sysfs with: [mount -t sysfs sysfs /tmp/68291840/sys]
arm: running the provision hook
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 Unable to load communicator config from state to populate provisionHookData
2022/11/26 09:08:04 [INFO] (telemetry) Starting provisioner shell
==> arm: Provisioning with shell script: /tmp/packer-shell526167725
2022/11/26 09:08:04 packer-provisioner-shell plugin: Opening /tmp/packer-shell526167725 for reading
2022/11/26 09:08:04 packer-provisioner-shell plugin: [INFO] 38 bytes written for 'uploadData'
2022/11/26 09:08:04 [INFO] 38 bytes written for 'uploadData'
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 Uploading to chroot dir: /tmp/68291840/tmp/script_9792.sh
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 Executing: /bin/sh []string{"/bin/sh", "-c", "chroot /tmp/68291840 /bin/sh -c \"chmod 0755 /tmp/script_9792.sh\""}
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 Chroot execution exited with '127': '"chmod 0755 /tmp/script_9792.sh"'
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 [INFO] RPC endpoint: Communicator ended with: 127
2022/11/26 09:08:04 [INFO] RPC client: Communicator ended with: 127
2022/11/26 09:08:04 [INFO] RPC endpoint: Communicator ended with: 127
2022/11/26 09:08:04 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 127
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 Executing: /bin/sh []string{"/bin/sh", "-c", "chroot /tmp/68291840 /bin/sh -c \"chmod +x /tmp/script_9792.sh; PACKER_BUILDER_TYPE='arm' PACKER_BUILD_NAME='arm' /tmp/script_9792.sh\""}
==> arm: -c: 0: Can't open chmod +x /tmp/script_9792.sh; PACKER_BUILDER_TYPE='arm' PACKER_BUILD_NAME='arm' /tmp/script_9792.sh
2022/11/26 09:08:04 [INFO] 0 bytes written for 'stdout'
2022/11/26 09:08:04 [INFO] 119 bytes written for 'stderr'
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 Chroot execution exited with '127': '"chmod +x /tmp/script_9792.sh; PACKER_BUILDER_TYPE='arm' PACKER_BUILD_NAME='arm' /tmp/script_9792.sh"'
2022/11/26 09:08:04 packer-builder-arm plugin: 2022/11/26 09:08:04 [INFO] RPC endpoint: Communicator ended with: 127
2022/11/26 09:08:04 [INFO] RPC client: Communicator ended with: 127
2022/11/26 09:08:04 [INFO] RPC endpoint: Communicator ended with: 127
2022/11/26 09:08:04 packer-provisioner-shell plugin: [INFO] 0 bytes written for 'stdout'
2022/11/26 09:08:04 packer-provisioner-shell plugin: [INFO] 119 bytes written for 'stderr'
2022/11/26 09:08:04 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 127
2022/11/26 09:08:04 [INFO] (telemetry) ending shell
==> arm: error while running provision hook: Script exited with non-zero exit status: 127. Allowed exit codes are: [0]
arm: optional (please ignore) `fuser -k` failed with exit status 1:
arm: unmounting /tmp/68291840/sys
arm: unmounting /tmp/68291840/proc/sys/fs/binfmt_misc
arm: unmounting /tmp/68291840/proc
arm: unmounting /tmp/68291840/dev/pts
arm: unmounting /tmp/68291840/dev
arm: unmounting /tmp/68291840/boot
arm: unmounting /tmp/68291840
2022/11/26 09:08:05 [INFO] (telemetry) ending arm
Build 'arm' errored after 1 minute 34 seconds: build was halted
==> Wait completed after 1 minute 34 seconds
==> Some builds didn't complete successfully and had errors:
--> arm: build was halted
==> Builds finished but no artifacts were created.
==> Wait completed after 1 minute 34 seconds
2022/11/26 09:08:05 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2022/11/26 09:08:05 machine readable: arm,error []string{"build was halted"}
==> Builds finished but no artifacts were created.
2022/11/26 09:08:05 [INFO] (telemetry) Finalizing.
2022/11/26 09:08:05 waiting for all plugin processes to complete...
2022/11/26 09:08:05 /usr/bin/packer-builder-arm: plugin process exited
2022/11/26 09:08:05 /usr/bin/packer: plugin process exited
Local env:
uname -a
Linux debian 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux
docker --version
Docker version 20.10.21, build baeda1f
Another example with a similar error using a different RPi image:
https://github.com/openoms/joininbox/actions/runs/3553055263/jobs/5968353247
Any suggestion how to get past by this?
The workflow was working last 2 months ago: https://github.com/openoms/joininbox/actions/runs/3214574185
Related tests in PR: openoms/joininbox#103
@openoms Tried to build your PR openoms/joininbox#103 and https://github.com/mkaczanowski/packer-builder-arm/blob/master/boards/raspberry-pi/raspbian.json locally with the 1.0.4 version and that works in both cases... see the suggestion in your PR. Can you commit that and post if it worked? thanks
hm, tested in this replicated PR and fails dbast/joininbox#1 .... despite working locally ... needs more investigation
@openoms Got it building in dbast/joininbox#1 :)
The solution was to add as step (even though that should not be required as the container contains the qemu stuff)
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
with:
platforms: linux/amd64,linux/arm64
And also the switch from /usr/bin/qemu-arm-static to /usr/bin/qemu-aarch64-static as done in the PR seems to be required. Created a PR that should build after being approved openoms/joininbox#106
Not sure yet what is causing this. Currently comparing various build logs and code snippets:
- https://github.com/tonistiigi/binfmt/blob/master/Dockerfile#L126-L134
- https://github.com/mkaczanowski/packer-builder-arm/blob/master/docker/Dockerfile#L52
- https://github.com/mkaczanowski/packer-builder-arm/blob/master/docker/entrypoint.sh#L7
Ideas to solve this differently welcome.
Created #246 for further investigations and to have this case also tested by the CI of the repo.
With your help this issue is fixed in openoms/joininbox#103. Thank you so much! Closing.