[Shamsidinkhon]

Gazelle calls GenerateRules for one directory at a time, and the plugin parses only the files of that directory, at most 6 at a time. Most directories have only a few Python files, so on a large repository about 2 CPUs are busy however many the machine has, and parsing takes most of the run time.

This adds a prefetcher that parses files ahead of the walk. When Configure reaches a directory passed on the command line, the prefetcher lists the Python files under it from the walk's cached directory listings (walk.GetDirInfo), honoring -r=false, and parses them in the background in the order rules will be generated, at most GOMAXPROCS at a time. When GenerateRules parses a file, it takes the prefetched result, waiting for it if needed; a file that wasn't prefetched is parsed as before. Only the timing of parsing changes, so the generated BUILD files are the same. The prefetcher is only set up for commands with a -mode flag (update and fix). It is a package-level variable because CheckFlags, Configure and the parser share no other state; it could move into the Configurer if preferred.

I wrote this change with an AI coding assistant (Claude Code) and reviewed it myself. I'll read and answer review comments personally; no agent posts on this PR.

[aignas]

Please add unit tests for the new code.

[Shamsidinkhon]

Please add unit tests for the new code.

Thanks, I added unit tests

[dougthor42]

I ran this on our relatively large repo (~7500 targets, ~7700 python files, ~1940 dirs).

Overall speedup was ~76s --> 8.6s, an 88.7% reduction in overall time, hot damn!

Granted, this was on a 64-core cloud VM...

Before

On rules_python 2.5.0

$ hyperfine --warmup=1 --runs=3 'bazel run --bes_backend= //:gazelle'
Benchmark 1: bazel run --bes_backend= //:gazelle
  Time (mean ± σ):     76.605 s ±  0.107 s    [User: 76.476 s, System: 4.481 s]
  Range (min … max):   76.485 s … 76.691 s    3 runs

After

On commit db95bd53

$ hyperfine --warmup=1 --runs=3 'bazel run --bes_backend= //:gazelle'
Benchmark 1: bazel run --bes_backend= //:gazelle
  Time (mean ± σ):      8.611 s ±  0.550 s    [User: 125.623 s, System: 6.973 s]
  Range (min … max):    8.224 s …  9.241 s    3 runs

[dougthor42]

IIRC we no longer need these notices.

[dougthor42]

Thoughts on guarding things behind an env var or CLI flag?

My main concern is edge cases, where it would be nice to have an easy way to go back to the prior behavior to double-check things.