Skip to content

perf: import-profiler redundantly reads and counts module lines on every iteration #17792

Description

@hebaalazzeh

Description

In scripts/import_profiler/profiler.py, the worker process (run_worker) scans and counts the lines of every imported .py file across all loaded modules after importing the target module:

    loaded_lines = 0
    for m in new_modules:
        try:
            file_path = sys.modules[m].__file__
            ...
            if file_path.endswith('.py'):
                with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
                    loaded_lines += sum(1 for _ in f)

For each package tested, the master process executes 11 iterations for the baseline commit and 11 iterations for the PR commit (22 iterations total). Currently, every single one of those 22 iterations opens, reads, and counts the lines of all loaded files.

Because the code volume stats (loaded modules and lines) are deterministic, this results in significant redundant disk I/O and CPU overhead.

Impact / Reproduction Example

When profiling a large package like google-cloud-compute:

Loaded Modules: 1,414 modules
Loaded Lines: 1,056,448 lines of code
Line Counting overhead: ~310 ms per worker process iteration.
Total Overhead: Across 22 iterations, the profiler spends ~6.8 seconds purely opening and reading Python files to count lines.

Proposed Solution

Optimize the script to count loaded lines only on the first iteration, and skip this calculation on subsequent runs by passing an internal --skip-line-count flag to worker subprocesses. This reduces the line-counting overhead from ~6.8 seconds to ~0.6 seconds per package run.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions