Skip to content

inspect.getsourcelines() is buggy on 3.12 #104866

Description

@JelleZijlstra
% cat gsl.py
import inspect

def test_list_add(self):
    def capybara() -> None:
        assert_is_value(
            [x] + [y],
            z,
        )
        assert_is_value(
            # in the binop implementation
            a,
        )

print(inspect.getsourcelines(test_list_add))
% python3.11 gsl.py 
(['def test_list_add(self):\n', '    def capybara() -> None:\n', '        assert_is_value(\n', '            [x] + [y],\n', '            z,\n', '        )\n', '        assert_is_value(\n', '            # in the binop implementation\n', '            a,\n', '        )\n'], 3)
% ~/py/cpython/python.exe gsl.py 
(['def test_list_add(self):\n', '    def capybara() -> None:\n', '        assert_is_value(\n', '            [x] + [y],\n', '            z,\n', '        )\n', '        assert_is_value(\n', '            # in the binop implementation\n'], 3)
% ~/py/cpython/python.exe -V
Python 3.12.0b1+

On 3.12, the last two lines (a, )) are not returned as part of the source lines. When I tried to minify by removing the first assert_is_value() call, it instead returned the print() line as part of the function.

This looks related to the tokenizer, cc @pablogsal @lysnikolaou.

Linked PRs

Activity

added
type-bugAn unexpected behavior, bug, or error
3.12only security fixes
3.13only security fixes
on May 24, 2023
added
stdlibStandard Library Python modules in the Lib/ directory
and removed
stdlibStandard Library Python modules in the Lib/ directory
on May 24, 2023

lysnikolaou commented on May 24, 2023

@lysnikolaou
Member

This is a bug in the implementation of the new tokenize module. The tokenizer checks whether something is in a parenthesized block in order to emit a NL token instead of a NEWLINE token. It also sets and unsets a flag about a comment-only line, in order to do the same. When both are true (a comment inside a parenthesized block), it nevers resets the comment flag. Here's a minimal example:

cpython onmain [$?] via C v14.0.3-clang via 🐍 pyenv 3.11.3
❯ cat hello.py              
[
    # hello
    1
]

cpython onmain [$?] via C v14.0.3-clang via 🐍 pyenv 3.11.3
❯ ./python.exe -m tokenize hello.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,1:            OP             '['            
1,1-1,2:            NL             '\n'           
2,4-2,11:           COMMENT        '# hello'      
2,11-2,12:          NL             '\n'           
3,4-3,5:            NUMBER         '1'            
3,5-3,6:            NL             '\n'           
4,0-4,1:            OP             ']'            
4,1-4,2:            NL             '\n'       # This should be a NEWLINE instead    
5,0-5,0:            ENDMARKER      ''             

I'm opening a PR shortly.

added a commit that references this issue on May 24, 2023
added a commit that references this issue on May 24, 2023
added a commit that references this issue on May 24, 2023
added a commit that references this issue on May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixestype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions