Description
In scripts/import_profiler/profiler.py, the worker process (run_worker) scans and counts the lines of every imported .py file across all loaded modules after importing the target module:
loaded_lines = 0
for m in new_modules:
try:
file_path = sys.modules[m].__file__
...
if file_path.endswith('.py'):
with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
loaded_lines += sum(1 for _ in f)
For each package tested, the master process executes 11 iterations for the baseline commit and 11 iterations for the PR commit (22 iterations total). Currently, every single one of those 22 iterations opens, reads, and counts the lines of all loaded files.
Because the code volume stats (loaded modules and lines) are deterministic, this results in significant redundant disk I/O and CPU overhead.
Impact / Reproduction Example
When profiling a large package like google-cloud-compute:
Loaded Modules: 1,414 modules
Loaded Lines: 1,056,448 lines of code
Line Counting overhead: ~310 ms per worker process iteration.
Total Overhead: Across 22 iterations, the profiler spends ~6.8 seconds purely opening and reading Python files to count lines.
Proposed Solution
Optimize the script to count loaded lines only on the first iteration, and skip this calculation on subsequent runs by passing an internal --skip-line-count flag to worker subprocesses. This reduces the line-counting overhead from ~6.8 seconds to ~0.6 seconds per package run.
Description
In
scripts/import_profiler/profiler.py, the worker process (run_worker) scans and counts the lines of every imported.pyfile across all loaded modules after importing the target module:For each package tested, the master process executes 11 iterations for the baseline commit and 11 iterations for the PR commit (22 iterations total). Currently, every single one of those 22 iterations opens, reads, and counts the lines of all loaded files.
Because the code volume stats (loaded modules and lines) are deterministic, this results in significant redundant disk I/O and CPU overhead.
Impact / Reproduction Example
When profiling a large package like google-cloud-compute:
Loaded Modules: 1,414 modules
Loaded Lines: 1,056,448 lines of code
Line Counting overhead: ~310 ms per worker process iteration.
Total Overhead: Across 22 iterations, the profiler spends ~6.8 seconds purely opening and reading Python files to count lines.
Proposed Solution
Optimize the script to count loaded lines only on the first iteration, and skip this calculation on subsequent runs by passing an internal --skip-line-count flag to worker subprocesses. This reduces the line-counting overhead from ~6.8 seconds to ~0.6 seconds per package run.