awk detection is broken on Musl-based Linux distros

Currently I test only the OpenMPTCProuter distro, but OpenWRT probably suffers in the same way.

The distro provides BusyBox version of AWK which seems to have a mix of features from nawk, gawk and probably some other implementations too: https://wiki.alpinelinux.org/wiki/Awk

My idea was that the boostrap/bootstrap script should either pick up the OS-native version (/usr/bin/awk -> ../../bin/busybox) or bootstrap its' own version (/usr/pkg/bin/nawk).

The bootstrapping seems to to ignore the former and require the latter. The problem is that the latter isn't built yet at the moment it gets required:

===> bootstrap command: bootstrap/bootstrap 
===> bootstrap started: Sun Mar 24 21:02:38 UTC 2024
===> running: /bin/sed -e 's|@DEFAULT_INSTALL_MODE@|'0755'|' /root/debug/src/github.com/NetBSD/pkgsrc.git/sysutils/install-sh/files/install-sh.in > /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh
===> running: /bin/chmod +x /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh
===> Creating default mk.conf in /root/debug/src/github.com/NetBSD/pkgsrc.git/work
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/share/mk
===> Bootstrapping mk-files
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/pkgtools/bootstrap-mk-files/files && env CP=/bin/cp  OPSYS=Linux MK_DST=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/share/mk ROOT_GROUP=root ROOT_USER=root SED=/bin/sed SYSCONFDIR=/usr/pkg/etc /bin/sh ./bootstrap.sh)
===> Bootstrapping bmake
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh configure  --prefix=/root/debug/src/github.com/NetBSD/pkgsrc.git/work --with-default-sys-path=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/share/mk --with-machine-arch=x86_64 )
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh make-bootstrap.sh)
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake
===> Building libnbcompat
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /bin/sh ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
===> Bootstrapping pkgtools
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive; env  CONFIG_SHELL=/bin/sh /bin/sh ./configure  --enable-static --disable-shared --disable-bsdtar --disable-bsdcpio --disable-bsdunzip --disable-posix-regex-lib --disable-xattr --disable-maintainer-mode --disable-acl --without-zlib --without-bz2lib --without-iconv --without-lzma --without-lzo2 --without-lz4 --without-nettle --without-openssl --without-xml2 --without-expat --without-zstd MAKE=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install; env  CPPFLAGS=' -I/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat -I/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive/libarchive' LDFLAGS=' -L/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat' LIBS=' -lnbcompat' /bin/sh ./configure  -C --enable-bootstrap --prefix=/usr/pkg --sysconfdir=/usr/pkg/etc --with-pkgdbdir=/usr/pkg/pkgdb --infodir=/usr/pkg/info --mandir=/usr/pkg/man  && STATIC_LIBARCHIVE=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive/.libs/libarchive.a STATIC_LIBARCHIVE_LDADD='' PKGSRC_MACHINE_ARCH=x86_64 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/add/pkg_add /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_add
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/admin/pkg_admin /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_admin
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/create/pkg_create /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_create
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/info/pkg_info /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_info
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: {print $3; exit}: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/tools/gmake.mk" line 43: warning: "/usr/bin/make --version |				  '{print $3; exit}'" returned non-zero status
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/pkgtools/cwrappers && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  MAKE_JOBS=1 PKG_COMPRESSION=none PKGSRC_KEEP_BIN_PKGS=no MAKECONF=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/mk.conf install)
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: {print $3; exit}: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/tools/gmake.mk" line 43: warning: "/usr/bin/make --version |				  '{print $3; exit}'" returned non-zero status
/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/pkgformat/pkg/list-dependencies: line 47: /usr/pkg/bin/nawk: not found
===> Installing dependencies for cwrappers-20220403
/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/pkgformat/pkg/list-dependencies: line 47: /usr/pkg/bin/nawk: not found
*** Error code 1

Stop.
bmake: stopped in /root/debug/src/github.com/NetBSD/pkgsrc.git/pkgtools/cwrappers
===> exited with status 1
aborted.

(I trimmed down details from all earlier steps except for the last three ones)

@jperkin

@jperkin responded on IRC on Sunday, March 17th, 2024:

I'd start from first-principles

firstly, does it not exist, or is /usr/pkg/bin/nawk there but somehow built with an invalid interpreter or something?

nawk is only bootstrapped if a native awk isn't found or is unsuitable, so I'd also check to see why your native awk isn't sufficient and if it can be provided/fixed instead

then if it is required and is missing, I'd be looking at the bootstrap log to figure out why it failed and how that wasn't reported as a failure to the bootstrap script

as this is Linux you'll be looking at mk/tools/tools.Linux.mk for how the native awk is found

To be continued.

Just to document my latest finding. It looks like the path for AWK is decided in mk/tools/replace.mk:

    226 .if !defined(TOOLS_IGNORE.awk) && !empty(_USE_TOOLS:Mawk)
    227 .  if ${PKGPATH} == lang/nawk
    228 MAKEFLAGS+=                     TOOLS_IGNORE.awk=
    229 .  elif !empty(_TOOLS_USE_PKGSRC.awk:M[yY][eE][sS])
    230 TOOLS_DEPENDS.awk?=             nawk>=20040207:../../lang/nawk
    231 TOOLS_CREATE+=                  awk
    232 TOOLS_PATH.awk=                 ${TOOLBASE}/bin/nawk
    233 .  endif
    234 .endif

... with TOOLBASE being set in mk/bsd.prefs.mk:

    607 LOCALBASE?=             /usr/pkg
    608 TOOLBASE:=              ${LOCALBASE}

remake: update to version 4.3+dbg-1.6

It seems mk/tools/tools.Linux.mk tries to find "gawk" specifically, and otherwise will not assign anything. That whole file basically assumes GNU tooling. (You kind of already partly answered your own question there in #130 (comment) .)

A dependency may indeed be specified in mk/tools/replace.mk where you quote, in certain contexts only.

(If you're looking for a single place where everything is revealed, well, a lot of the pkgsrc infrastructure isn't like that. It's more complicated.)

lang/pear: update to 1.10.15

@jperkin

Confirmed on the following distros, attaching the logs inline:

Quoting @jperkin's IRC response and inviting for assistance:

firstly, does it not exist, or is /usr/pkg/bin/nawk there but somehow built with an invalid interpreter or something?

Surely the file does not exist yet:

# ls /usr/pkg/
ls: /usr/pkg/: No such file or directory

nawk is only bootstrapped if a native awk isn't found or is unsuitable, so I'd also check to see why your native awk isn't sufficient and if it can be provided/fixed instead

My impression is that nawk was not bootstrapped. That's the root problem I see.

then if it is required and is missing, I'd be looking at the bootstrap log to figure out why it failed and how that wasn't reported as a failure to the bootstrap script

Jonathan, I still doesn't understand that part. Sounds too abstract, maybe even ambiguous to me.
A guidance is needed :)

as this is Linux you'll be looking at mk/tools/tools.Linux.mk for how the native awk is found

The only awk references I found in mk/tools/tools.Linux.mk are:

    105 .  if exists(${_path}/gawk)
    106 TOOLS_PLATFORM.gawk?=           ${_path}/gawk
    107 .  endif
    ...
    280
    281 # assuming grep/sed/tar/awk are usually the GNU versions, is this safe?
    282
    283 .if !empty(TOOLS_PLATFORM.bison)
    284 TOOLS_PLATFORM.bison-yacc?=     ${TOOLS_PLATFORM.bison} -y
    285 .endif
    286
    287 TOOLS_PLATFORM.awk?=            ${TOOLS_PLATFORM.gawk}
    288

Which @dhgutteridge explained above:

It seems mk/tools/tools.Linux.mk tries to find "gawk" specifically, and otherwise will not assign anything. That whole file basically assumes GNU tooling.

Debian 12 also provides non-GNU implementation by default, the mawk. Interestingly, pkgsrc decides to bootstrap its' own nawk here just fine resulting in /usr/pkg/bin/nawk.

From 05-bootstrap-builds-nawk-on-Debian-12.log:

--- /root/debug/src/github.com/NetBSD/pkgsrc.git/debug/logs/03-bootstrap-misses-nawk-on-Alpine-Linux.log
+++ /root/debug/src/github.com/NetBSD/pkgsrc.git/debug/logs/05-bootstrap-builds-nawk-on-Debian-12.log
@@ -1,16 +1,16 @@
 ===> bootstrap command: bootstrap/bootstrap
-===> bootstrap started: Fri Apr 19 20:26:12 EEST 2024
+===> bootstrap started: Sat Apr 20 08:29:37 AM EEST 2024
 Working directory is: /root/debug/src/github.com/NetBSD/pkgsrc.git/work
  ...
-===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh make-bootstrap.sh)
  ...
-===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake
+===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /usr/bin/sh make-bootstrap.sh)
  ...
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake
 ===> Building libnbcompat
-===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat
-===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /bin/sh ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat
+===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /usr/bin/sh ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
 configure: creating cache config.cache
-checking build system type... x86_64-pc-linux-musl
-checking host system type... x86_64-pc-linux-musl
+checking build system type... x86_64-pc-linux-gnu
+checking host system type... x86_64-pc-linux-gnu
 checking for gcc... gcc
 checking whether the C compiler works... yes
 checking for C compiler default output file name... a.out
@@ -265,13 +267,11 @@
 checking whether ln -s works... yes
 checking for ranlib... ranlib
 checking for gawk... no
-checking for mawk... no
-checking for nawk... no
-checking for awk... awk
+checking for mawk... mawk
 checking for ar... ar
 checking for fparseln in -lutil... no
-checking for grep that handles long lines and -e... /bin/grep
-checking for egrep... /bin/grep -E
+checking for grep that handles long lines and -e... /usr/bin/grep
+checking for egrep... /usr/bin/grep -E
 checking for _Bool... yes
 checking for stdbool.h that conforms to C99... yes
 checking for alloca.h... yes
  ...
 ranlib libnbcompat.a
+===> Bootstrapping awk
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk
+===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1 -f Makefile CC="cc" CFLAGS="")
+cc   -c awkgram.tab.c
+cc   -c b.c
+cc   -c main.c
+cc   -c parse.c
+cc   -c proctab.c
+cc   -c tran.c
+cc   -c lib.c
+cc   -c run.c
+cc   -c lex.c
+cc  awkgram.tab.o b.o main.o parse.o proctab.o tran.o lib.o run.o lex.o   -lm -o a.out
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk/a.out /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/nawk
 ===> Bootstrapping pkgtools

I wonder what makes the bootstrap fail to reach (or just skip) this step on the topic distros.

OK, found the answer in bootstrap/bootstrap:

 737         # Debian/Ubuntu's awk is mawk, and mawk does not understand
 738         # some regexps used in pkgsrc/mk.
 739         if [ -f /etc/debian_version ]; then
 740                 need_awk=yes

So it's tied to the hardcoded OS name. Waiting for the suggestions on how to proceed in a clean way :)

devel/hs-async: Update to 2.2.5

textproc/hs-commonmark-extensions: Update to 0.2.5.4

textproc/hs-commonmark-pandoc: Update to 0.2.2.1

uriparser: update to 0.9.8.

@sskras

So it's tied to the hardcoded OS name. Waiting for the suggestions on how to proceed in a clean way :)

The main thing we would avoid changing (active users and all...) is Enterprise Linux. Perhaps you can tell us if GNU awk is specifically identifiable on distributions like CentOS @sskras (perhaps installed as "gawk" in some directory)? Otherwise we can set need_awk=yes on all Linux...

editors/reovim: update to 0.8.1

libsidplayfp: update to 2.16.0.

devel/ruby-curses: update to 1.6.0

xfce4-screenshooter: update to 1.11.3

filesystems/fuse-bindfs: update to 1.18.4

devel/ruby-marcel: update to 1.2.1

p5-XML-LibXML: update to 2.0213.

ocaml-zarith: update to 1.14

finance/rex: update to 0.2.9

net/xfr: update to 0.9.21

bgpq4: Add version 1.16

misc/rozi: update to 0.0.31

devel/gitpane: update to 0.18.4

Navigation Menu

Repository navigation

awk detection is broken on Musl-based Linux distros #132

113 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

awk detection is broken on Musl-based Linux distros #132

Description

Activity

sskras commented on Mar 25, 2024

sskras commented on Mar 25, 2024

dhgutteridge commented on Mar 27, 2024

sskras commented on Apr 20, 2024

sskras commented on Apr 20, 2024

sskras commented on Apr 20, 2024

alarixnia commented on May 15, 2024

113 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions