Skip to content

gh-153568: Reduce parser allocations - #157374

Open
pablogsal wants to merge 7 commits into
python:mainfrom
pablogsal:gh-153568-parser-allocations
Open

gh-153568: Reduce parser allocations#157374
pablogsal wants to merge 7 commits into
python:mainfrom
pablogsal:gh-153568-parser-allocations

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 12, 2026

Copy link
Copy Markdown
Member

Allocate tokens in blocks and keep text only for names, literals and type comments. Use a bit for the Barry spelling and put the keyword error messages in the grammar. Create the identifier cache on first use and reuse the newline buffer. Reuse parsed names and numbers during backtracking, and reset memo filters before the error pass.

Related to #153568.

Codebase Python files Parsing speed Peak parser memory
Standard library 2,060 8.0% faster 11.1% less
Home Assistant 18,731 4.4% faster 6.7% less
mypy 442 5.9% faster 10.7% less
PyTorch 4,890 7.2% faster 5.1% less

@pablogsal
pablogsal force-pushed the gh-153568-parser-allocations branch from 9f8c9c0 to 0cd6ee2 Compare September 12, 2026 22:46
@pablogsal

Copy link
Copy Markdown
Member Author

This looks gigantic because the parser autogenerated changes but it's quite tiny :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant