Repository navigation
curses "insstr" does not handle newline properly on Mac OS #107267
Description
Activity
How did you install Python (python.org installer, homebrew, ...)?
I get the output in the second screenshot when using Python 3.10.11 or 3.11.4 on macOS 13.4 when using the installer on python.org.
Thanks! I installed python on both Mac and Ubuntu using "pyenv". Maybe that's the problem? I'll try using a homebrew installed Python on my Mac.
I just tried on my M2 Mac. I have homebrew installed Python 3.9.17, 3.10.12, 3.11.4, and the OS default installed 3.9.6. e.g.,:
$ /usr/bin/python3 --version Python 3.9.6 $ /usr/bin/python3 curses_bug.py # This uses the default OS python 3.9.6, where curses_bug.py is the above example.All had (incorrect) results as in the first screenshot.
This could be due to the particular version of the ncurses library used by the CPython extension module.
The script below prints the version of ncurses, for me this prints
b'ncurses 5.9.20120616'.import curses import ctypes fn = ctypes.CDLL(None).curses_version fn.restype = ctypes.c_char_p print(fn())
Also: What's the value of the environment variable
TERMon your system? For me it isxterm-256color(which should be the default for Terminal.app on macOS).And finally, could you check which ncurses library your Python build links to. The following command should show this information:
$ otool -vL $(python3.10 -c 'import _curses; print(_curses.__file__)') /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_curses.cpython-310-darwin.so (architecture x86_64): /Library/Frameworks/Python.framework/Versions/3.10/lib/libncursesw.5.dylib (compatibility version 5.0.0, current version 5.0.0) time stamp 2 Thu Jan 1 01:00:02 1970 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0) time stamp 2 Thu Jan 1 01:00:02 1970 /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_curses.cpython-310-darwin.so (architecture arm64): /Library/Frameworks/Python.framework/Versions/3.10/lib/libncursesw.5.dylib (compatibility version 5.0.0, current version 5.0.0) time stamp 2 Thu Jan 1 01:00:02 1970 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0) time stamp 2 Thu Jan 1 01:00:02 1970
On my machine this shows that
_cursesis linked to a copy of ncurses shipped with our installer. The system copy of ncurses is a lot older (b'ncurses 5.7.20081102') and might contain bugs.Thank you for your comments.
My
TERMvariable isxterm-256color(in both iTerm and Terminal.app).Here is the output of the little Python program you provided:
Python 3.11.3 (main, May 13 2023, 16:34:49) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import curses >>> import ctypes >>> fn = ctypes.CDLL(None).curses_version >>> fn.restype = ctypes.c_char_p >>> print(fn()) b'ncurses 5.7.20081102'
And here is the output of the
otoolcommand:$ otool -vL $(python -c 'import _curses; print(_curses.__file__)') ~/.pyenv/versions/3.11.3/lib/python3.11/lib-dynload/_curses.cpython-311-darwin.so: /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0) time stamp 2 Wed Dec 31 17:00:02 1969 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3) time stamp 2 Wed Dec 31 17:00:02 1969
So it looks like the
pyenvinstalled (and also homebrew installed) Python links to the system provided ncurses library (I think it is from the xcode command line tools installation).I think when I do
pyenv install 3.11.3. It downloaded the source and built Python locally on my machine. Maybe I should report the problem to pyenv? so they are aware of the dated ncurses library?However as I mentioned, I also tried a C program:
#include <curses.h> int main() { WINDOW* scr = initscr(); waddstr(scr, "first line\n"); waddstr(scr, "second line"); getch(); mvwinsstr(scr, 0, 5, "X\nXX"); getch(); endwin(); }
Call this file
curses_bug.c:$ gcc --version Apple clang version 14.0.3 (clang-1403.0.22.14.1) Target: arm64-apple-darwin22.5.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin $ gcc curses_bug.c -Wall -o curses_bug -lncurses $ otool -vL ./curses_bug ./curses_bug: /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0) time stamp 2 Wed Dec 31 17:00:02 1969 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3) time stamp 2 Wed Dec 31 17:00:02 1969
It appears the C program is linking with the same ncurses library as the pyenv installed Python 3.11.3. And yet this C program behaves correctly (i.e., as in the second screenshot I provided).
I checked out the
cpythongit repository and did a local build on my Mac.The first build (inside the
cpythonrepository directory):$ ./configure $ make $ ./python.exe
Then:
Python 3.13.0a0 (heads/main:810d5d87d9, Jul 29 2023, 00:40:02) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import curses >>> import ctypes >>> fn = ctypes.CDLL(None).curses_version >>> fn.restype = ctypes.c_char_p >>> print(fn()) b'ncurses 5.7.20081102'
otool -vL $(./python.exe -c 'import _curses; print(_curses.__file__)') ~/tmp/cpython/build/lib.macosx-13.4-arm64-3.13/_curses.cpython-313-darwin.so: /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0) time stamp 2 Wed Dec 31 17:00:02 1969 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3) time stamp 2 Wed Dec 31 17:00:02 1969
So it is the same as before and it still uses the system
ncurseslibrary. However the above small program I created withinsstrnow behaves correctly (results as shown in the second screenshot).My second build:
$ make distclean $ git checkout 3.12 $ ./configure $ make $ ./python.exe Python 3.12.0b4+ (heads/3.12:80aebd54c8, Jul 29 2023, 01:00:29) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
The rest of the outputs are the same as above (still uses the system
ncurses). And this version also behaves correctly on the littleinsstrtest program.Third build:
$ make distclean $ git checkout 3.11 $ ./configure $ make $ ./python.exe Python 3.11.4+ (heads/3.11:4049c5d6f8, Jul 29 2023, 01:15:27) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
I am skipping the other outputs as they are still the same (still uses the system
ncurseslibrary). However this version behaves incorrectly on the littleinsstrtest program. It behaves exactly as mypyenvinstalled Python 3.11.3.So it seems something has happened between 3.11 and 3.12 and that caused the different behavior on the
insstrmethod in thecursesmodule.@ned-deily and @erlend-aasland, any idea what might be going on here?
In summary: there appears to be something wrong with the curses extension module in 3.11 but not in 3.12 when linking with the system install of ncurses. Out installers work fine (but link with our own copy of that library). A small C program linking with the system install works fine.
There are some differences between 3.10 and 3.11 in _cursesmodule, but none seem relevant (3.12...3.10).
Reacted by Erlend E. Aasland@yzhanglbto : thanks for your extensive testing, this should make it easier to pinpoint the problem.
After a quick look at this, I think the difference in behavior is due to a difference between 3.11 and 3.12 in how the Python
readlinemodule is being built.In 3.11 and earlier, there was a bunch of code in
setup.pyto determine whether readline should try to link with a curses library or not. When building 3.11 from source, without a newer version of ncursesw like the python.org binaries do, the resulting readline module is linked with the old Apple-supplied ncurses. However, for 3.12 (and 3.13), it appears that the new replacement forsetup.pyis not choosing to linkreadlinewithncursesand, in this case, that makes the test pass either when building from source or when using the python.org binaries. I'm not at all familiar with the interdependencies of ourreadlinemodule,libeditor GNUreadline, and the variouscurses/ncurses/ncurseswlibs in use but I assume there was a good reason for attempting to linkreadlinewith the curses library when possible. So this is a difference in 3.12 that might be considered a bug. Another question is whether this change in behavior is only on macOS.@erlend-aasland, does this ring any bells?
For reference,
otool -Lof thereadlinemodules in the most recent python.org 3.11 and 3.12 releases:readline.cpython-311-darwin.so: /usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0) /Library/Frameworks/Python.framework/Versions/3.11/lib/libncursesw.5.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0) readline.cpython-312-darwin.so: /usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)[...] does this ring any bells?
@ned-deily, yes. Christian and I deliberately simplified some of the third-party dependency checks. I'll have a look at what we did and why we did it, but I won't be able to find time for it until a week or two.
Thanks for the heads-up, though!
I browsed through configure.ac regarding some other readline issues today, and I came across this comment, added by Christian in e925241 (gh-90005, gh-94452):
dnl NOTE: In the past we checked if readline needs an additional termcap
dnl library (tinfo ncursesw ncurses termcap). We now assume that libreadline
dnl or readline.pc provide correct linker information.fwiw I believe this may be a recent regression (and not in python?) -- perhaps with the macos updates to ncurses given the latest CVE
I updated from (unknown version) to 13.5.2 yesterday and this quirk breaks my text editor babi which seems to be exhibiting the same bug described here (but was working before I upgraded)
Since the last updates on this issue, it appears that Apple has provided some patches to the system version of ncurses on various older security releases of macOS and, FTR, as of the Python 3.14.0 alpha 5 pre-release and the upcoming 3.13.3 and 3.12.10 releases, python.org macOS installers now build and link with ncurses 6.5, the current stable release, and no longer with either ncurses 5.9 or with the Apple-supplied and -patched macOS system ncurses. There have also been some macOS-specific fixes in the Python curses module (for example, #111258). In any case, I am not able to reproduce the original problem on the current release of macOS (15.4 Sequoia) nor a couple of older versions (macOS Ventura 13.7.4 and Monterey 12.7.6). So let's close this issue now but feel free to reopen if the problem persists.
Bug report
Using Python 3.11.3 and 3.11.4 on Apple M2 Mac (Mac OS Ventura 13.4, iTerm and Terminal.app), Python 3.10.6 on Intel Mac (Mac OS Monterey 12.6, iTerm) produce:
Python 3.11.2 on Intel Ubuntu 22.04 (in GNOME terminal) produces correct behavior. An equivalent C program using ncurses directly on the above Macs (in the same terminal) also produces correct behavior.
