Skip to content

Exception in ucs2lib_utf8_encoder in _bootstrap_python #94526

Closed
Closed
Exception in ucs2lib_utf8_encoder in _bootstrap_python#94526
Labels
3.11only security fixes3.12only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Description

@pgy
pgy
opened on Jul 3, 2022

Bug report

I am building python in a directory that has non-ascii name on linux and get the following error:

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Exception ignored error evaluating path:
Traceback (most recent call last):
  File "<frozen getpath>", line 349, in <module>
ModuleNotFoundError: No module named 'encodings'
Fatal Python error: error evaluating path
Python runtime state: core initialized

Current thread 0x00007f16d6c10740 (most recent call first):
  <no Python frame>
make: *** [Makefile:1218: Python/frozen_modules/abc.h] Error 1

The problem seems to be that the limited environment used to execute the precompiled Modules/getpath.py does not have the encodings module.

I tracked the exception to a PyImport_ImportModule("encodings"); call in Python/codecs.c which is a consequence of a PyCodec_LookupError that happens in ucs2lib_utf8_encoder which I think gets called in getpath_dirname.

The original string is /home/pgy/letöltések/cpython/_bootstrap_python, and the unicode object ucs2lib_utf8_encoder gets as an argument is:

(rr) p PyObject_Print(unicode, stderr, 0)
'/home/pgy/let\udcc3\udcb6lt\udcc3\udca9sek/cpython/_bootstrap_python'

Your environment

  • CPython versions tested on: 7db1d2e
  • Operating system and architecture: Linux hostname 5.15.44-1-lts #1 SMP Mon, 30 May 2022 13:45:47 +0000 x86_64 GNU/Linux

Activity

  1. pgy
    added
    type-bugAn unexpected behavior, bug, or error
    on Jul 3, 2022
  2. kumaraditya303
    self-assigned this
    on Sep 17, 2022
  3. kumaraditya303
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    3.11only security fixes
    3.12only security fixes
    on Sep 17, 2022

  4. vstinner commented on Sep 29, 2022

    @vstinner
    vstinner
    on Sep 29, 2022
    Member

    I created PR #97645 to fix the root issue.

    I can reproduce the issue:

    • Clone Python Git repository in a non-ASCII directory: /home/vstinner/python/mainé (é is non-ASCII)
    • Build Python: ./configure --with-pydebug && make
    • make fails with: ModuleNotFoundError: No module named 'encodings'

    My locale encoding is UTF-8, my LC_CTYPE locale is fr_FR.UTF-8.

    The problem is that the getpath_dirname() function of Modules/getpath.c encodes the Unicode path to UTF-8/strict using s format and PyArg_ParseTuple(), whereas a path can contain surrogate characters.

    The getpath_dirname() and getpath_basename() convert their Unicode input string to bytes just to be able to use strrchr(path, SEP) since SEP is a bytes string. There is no need to convert these strings to bytes, whereas the conversion is the root issue here.

  5. vstinner
    added a commit that references this issue on Sep 30, 2022

    gh-94526: getpath_dirname() no longer encodes the path (#97645)

    9f2f1dd
  6. miss-islington
    added a commit that references this issue on Sep 30, 2022

    pythongh-94526: getpath_dirname() no longer encodes the path (pythonG…

    0aa7e74
  7. kumaraditya303
    removed their assignment
    on Sep 30, 2022

  8. vstinner commented on Sep 30, 2022

    @vstinner
    vstinner
    on Sep 30, 2022
    Member

    Fixed by PR #97645. Backport to 3.11: PR #97677.

  9. vstinner
    closed this as completedon Sep 30, 2022
  10. miss-islington
    added a commit that references this issue on Sep 30, 2022

    gh-94526: getpath_dirname() no longer encodes the path (GH-97645)

    6537bc9
  11. serhiy-storchaka
    added a commit that references this issue on Oct 2, 2022

    pythongh-94526: getpath_dirname() no longer encodes the path (python#…

    cc1e8e0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.11only security fixes3.12only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions