Repository navigation
Error while using module tokenize with a script file named token.py #95893
Description
Bug report
If you have a file called token.py and you try to use the module tokenize or any other module that uses it, like logging, the following error generates:
$ python -m tokenize token.py
Hello, World!
Traceback (most recent call last):
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/tokenize.py", line 35, in <module>
from token import EXACT_TOKEN_TYPES
ImportError: cannot import name 'EXACT_TOKEN_TYPES' from 'token' (/private/tmp/token.py)
$ cat token.py
def say_hello():
print("Hello, World!")
say_hello()$ python -c 'import logging'
Hello, World!
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/logging/__init__.py", line 26, in <module>
import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/traceback.py", line 5, in <module>
import linecache
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/linecache.py", line 11, in <module>
import tokenize
File "/Users/myuser/.pyenv/versions/3.9.4/lib/python3.9/tokenize.py", line 35, in <module>
from token import EXACT_TOKEN_TYPES
ImportError: cannot import name 'EXACT_TOKEN_TYPES' from 'token' (/private/tmp/token.py)
Your environment
$ python --version
Python 3.9.4
macOS Catalina
version 10.15.7
- CPython versions tested on:
- Operating system and architecture:
Activity
ronaldoussoren commentedon Aug 11, 2022
The explanation for this is the value of
sys.path: Forpython -m ...the current working directory is in early insys.path, which is why you'll get an error like this for all modules used by thetokenisemodule or during interpreter startup.The workaround is to place the file in a different directory (e.g.
python -m tokenize tmpdir/token.py).iritkatriel commentedon Aug 14, 2022
Another workaround is to avoid naming your modules with names that shadow stdlib modules. This is good practice - it is very confusing when something like ‘import collections’ import something different from what ypu would normally expect it to.
A common way to stumble upon this is with a local ‘test.py’ script.