Tune parallel parsing performance on macOS

[JukkaL]

Based on some quick experiments, the recently added paraller parsing may not perform well on macOS. Perform measurements and see if we can improve performance if there appears to be a macOS related bottleneck. Ideas to try include reusing the thread pool executor object and batching the files to parse.

[JukkaL]

I will look into this, cc @ilevkivskyi

[ilevkivskyi]

Oh, btw, I just figured out that I was only testing parallel parsing with parallel checking. In sequential mode there is a big non-parallelizeable chunk: deserializing the parsed ASTs. Extracting it may be quite awkward, but not impossible. It will improve performance for sequential mode (if we care about it).

[JukkaL]

I'm going to focus on parallel performance at first.

[JukkaL]

I did some initial experiments and measurements, and I think parallel parsing actually perhaps isn't the main issue, but get_source() seems pretty slow on macOS (at least on my work laptop, but I'll run some more experiments later). However, there should be no need to read the file in the Python process, as the native parser is reading the file anyway.

We could probably calculate the file hash and extract the inline mypy: config in Rust easily, for some extra parallelism. This would probably help on all platforms, though macOS might get the biggest benefit.

Next steps:

• Measure get_source() performance on both my personal mac and work mac, using compiled mypy.
• Measure it also on Linux (both in a VM and bare metal), using compiled mypy.

[ilevkivskyi]

@JukkaL

We could probably calculate the file hash and extract the inline mypy: config in Rust easily, for some extra parallelism

Yeah, that would be nice. Btw, looking at get_source() I think this part is broken with new parser:

    path = manager.maybe_swap_for_shadow_path(self.path)

i.e. options.shadow_file is effectively ignored, because new parser doesn't use the source, it uses the original path. Will you have time to handle this as well?

[JukkaL]

I created an issue to track the shadow file brokenness: #21222. I may have time to look at this, but probably not in the next few days.

[JukkaL]

I did some more analysis. get_source() looks like a somewhat significant bottleneck and worth fixing. The potential perf gain could be up to 5% on macOS. Batching parsed files could also improve performance on macOS, but the impact seem fairly minor (under 1%). So I'm planning to look into moving get_source() into the Rust extension first and then reassessing the impact of batching.

Reusing the thread pool executor doesn't seem important, since it doesn't get created that often.