gh-153568: Parse arithmetic with operator loops - #157413
Open
pablogsal wants to merge 2 commits into
Open
Conversation
Turn rules marked with operator_loop into a simple loop over operators and operands. Keep the cached result and the existing error rules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For
a + b + c, the parser first findsa, then calls the rule again to geta + b, then again to get(a + b) + c. It puts each partial result in the parse cache so the next call can pick it up. That works, but it adds rule calls, cache lookups and cache updates to something that is really just a loop.This adds an
(operator_loop)flag to the six arithmetic and bitwise rules. The generated code parsesa, keeps the current expression in a local variable, and loops over+ band+ c. Each step builds the next AST node directly. It only puts the finished expression in the cache, so all that bookkeeping between steps goes away.This also helps when there is no operator. Even a plain name passes through these rules, and the general parser still has to check whether it can grow the expression. The loop can just see that the next token isn't an operator and return.
The grammar and AST actions stay the same apart from the flag, and the existing error rules are still there.
The table compares parsing with and without these loops. Both builds include pending changes that reduce allocations, skip grammar choices using the next token, make stack checks and cache updates cheaper, and handle simple one-token expressions directly.
The timings include tokenization, building the AST and cleanup, without compiling bytecode or running the files.