Repository navigation
Docker build with GitHub Actions failed on multi arch with node:alpine #1798
Description
Activity
This also looks to be affecting another project: https://github.com/benphelps/homepage/actions/runs/3382149679
Going from
-FROM node:current-alpine +FROM docker.io/node:18-alpine
fixes it. Hard for me to tell if this is a problem with the image or Node at this point.
Reacted by Alicia Sykes and snowdreamI've stumbled upon the same issue in n-thumann/IPTV-ReStream#188.
It can be easily reproduced locally usingdocker run --rm arm32v7/node:19-alpine node -h. Simply printing the help hangs indefinitely for both arm32v6 and arm32v7, while any other architecture or prior Node.js release prints the help immediately.This may be an issue related to / in qemu, because running the same command on a natively on Alpine @ armv7 (Raspberry Pi in this case) doesn't cause any issues:
alpine:/tmp# uname -m armv7l alpine:/tmp# ./node -v v19.0.0 alpine:/tmp# ./node -h Usage: node [options] [ script.js ] [arguments] [... snip ...]I also tried to strace and noticed that
node -hyields a ton of[pid 52] mremap(0x8080c000, 4096, 8192, 0) = -1 ENOMEM (Out of memory)while hanging. Maybe someone else has an idea what to do here 🤔This is the same issue as #1794.
On Node 18 everything is OK, on Node 19 (
latest)npmsticks on some architectures. Seems to be upstream issue: npm/cli#5743- added a commit that references this issue
on Jun 5, 2023 Still reproduces on Node v20 images.
Reacted by Nicolas ThumannI did some investigation and it looks like this is caused by qemu, which is used internally in Docker to emulate foreign architectures, e.g. when running a ARMv7 binary on a x86_64 host.
This can be reproduced via:➜ ~ docker run -it --rm --platform linux/arm/v7 alpine:3.18.2 sh / # apk add nodejs-current=20.5.0-r0 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/armv7/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/armv7/APKINDEX.tar.gz (1/11) Installing ca-certificates (20230506-r0) (2/11) Installing libgcc (12.2.1_git20220924-r10) (3/11) Installing libstdc++ (12.2.1_git20220924-r10) (4/11) Installing ada-libs (2.5.1-r0) (5/11) Installing brotli-libs (1.0.9-r14) (6/11) Installing c-ares (1.19.1-r0) (7/11) Installing icu-data-en (73.2-r2) Executing icu-data-en-73.2-r2.post-install * * If you need ICU with non-English locales and legacy charset support, install * package icu-data-full. * (8/11) Installing icu-libs (73.2-r2) (9/11) Installing nghttp2-libs (1.55.1-r0) (10/11) Installing libuv (1.44.2-r2) (11/11) Installing nodejs-current (20.5.0-r0) Executing busybox-1.36.1-r0.trigger Executing ca-certificates-20230506-r0.trigger OK: 53 MiB in 26 packages / # wget https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/qemu-arm-8.0.3-r1.apk Connecting to dl-cdn.alpinelinux.org (146.75.118.132:443) saving to 'qemu-arm-8.0.3-r1.apk' qemu-arm-8.0.3-r1.ap 100% |*******************************************************************************************************************************************************| 1218k 0:00:00 ETA 'qemu-arm-8.0.3-r1.apk' saved / # / # apk add --allow-untrusted qemu-arm-8.0.3-r1.apk (1/1) Installing qemu-arm (8.0.3-r1) Executing busybox-1.36.1-r0.trigger OK: 56 MiB in 27 packages / # qemu-arm -version qemu-arm version 8.0.3 Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers / # /usr/bin/node -v v20.5.0 / # qemu-arm -strace /usr/bin/node -h [...] 55 open("/proc/meminfo",O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 17 55 fcntl64(17,F_SETFD,1) = 0 55 read(17,0x3fffeb2c,4095) = 1363 55 close(17) = 0 55 mremap(1073733632,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073729536,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073725440,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073721344,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073717248,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073713152,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073709056,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073704960,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) 55 mremap(1073700864,4096,8192,0,0,4096) = -1 errno=12 (Out of memory) [...]Reacted by Kai Ren and Kevin RohnThis seems to be indeed caused by a bug in qemu and is tracked in https://gitlab.com/qemu-project/qemu/-/issues/1729.
Reacted by Kai Ren, Kevin Rohn, mendhak and snowdream- added a commit that references this issue
on Nov 13, 2023 It seems that this is resolved with QEMU 8.1.0. There are still many
mremapsyscalls, but the same commands above complete when using https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/qemu-arm-8.1.3-r1.apk.Reacted by Koen KantersWith tonistiigi/binfmt#144 (QEMU 8.1.4) and
node:21it still doesn't work for me onarm32v6,arm32v7ands390xplatforms. Tried building on bothmacos-latestandubuntu-latestrunners:Reacted by snowdream- added a commit that references this issue
on Feb 10, 2024 I've tracked the issue some time now. Is it correct, that the root cause can be pinned down to the qemu bug? There does not appear to be anyone taking up the matter.
qemu-project/qemu#1729 has been closed recently, but the problem still occurs using latest image of
tonistiigi/binfmt(=qemu-v8.1.5) with currentnode:lts-alpine(=22.12.0-alpine3.21) andnode:alpine(=23.4.0-alpine3.21). Looks like the latest release of binfmt bundles qemu 8.1.5 which might be too old. This issue is probably depending on tonistiigi/binfmt#197 now.- added a commit that references this issue
on Jan 3, 2025 This is a stale issue.
Troubleshooting comments indicate that the issue was with Docker emulation.
This repo does supply emulation so cannot supply any fix if there are any outstanding issues. Newer releases of https://github.com/tonistiigi/binfmt may have resolved the issue.
Environment
Expected Behavior
The build is expected to complete without error, see: https://github.com/manuc66/node-hp-scan-to/actions/runs/3278864778/jobs/5397750728
Current Behavior
The build fail due to a timeout, see : https://github.com/manuc66/node-hp-scan-to/actions/runs/3304345713/jobs/5453599065
Possible Solution
I switched back to node:18-alpine and the build is working again, see: https://github.com/manuc66/node-hp-scan-to/actions/runs/3356995905/jobs/5562501857