Split type-checking into interface and implementation in parallel workers

[ilevkivskyi]

The general idea is very straightforward: when doing type-checking, we first type-check only module top-levels and those functions/methods that define/infer externally visible variables. Then we write cache and send new interface hash back to coordinator to unblock more SCCs early. This makes parallel type-checking ~25% faster.

However, this simple idea surfaced multiple quirks and old hacks. I address some of them in this PR, but I decided to handle the rest in follow up PR(s) to limit the size of this one.

First, important implementation details:

• On each select() call, coordinator collects all responses, both interface and implementation ones, and processes them as a single batch. This simplifies reasoning and shouldn't affect performance.
• We need to write indirect dependencies to a separate cache file, since they are only known after processing function bodies. I combine them together with error messages in files called foo.meta_ex.ff. Not 100% sure about the name, couldn't find anything more meaningful.
• Overload signatures are now processed as part of the top-level in type checker. This is a big change, but it is unavoidable and it didn't cause any problems with the daemon.
• Initializers (default values of function arguments) are now processed as part of the top-levels (to match runtime semantics). Btw @hauntsaninja you optimized them away in some cases, I am not sure this is safe in presence of walrus, see e.g. testWalrus.
• local_definitions() now do not yield methods of classes nested in functions. We add such methods to both symbol table of their actual class, and to the module top-level symbol table, thus causing double-processing.

Now some smaller things I already fixed:

• We used to have three scoping systems to track current class in type checker. One existed purely for the purpose of TypeForm support. I think two is enough, so I deleted the last one.
• AwaitableGenerator return type wrapping used to happen during processing of function body, which is obviously wrong.
• Invalid function redefinitions sometimes caused duplicate errors in case of partial types/deferrals. Now they should not, as I explicitly skip them after emitting first error.
• Some generated methods were not marked as such. Now they are.

Finally, some remaining problems and how I propose to address them in followups:

• Narrowing of final global variables is not preserved in functions anymore, see testNarrowingOfFinalPersistsInFunctions. Supporting this will be tricky/expensive, it would require preserving binder state at the point of each function definition, and restoring it later. IMO this is a relatively niche edge case, and we can simply "un-support" it (there is a simple workaround, add an assert in function body). To be clear, there are no problems with a much more common use of this feature: preserving narrowing in nested functions/lambdas.
• Support for --disallow-incomplete-defs in plugins doesn't work, see testDisallowIncompleteDefsAttrsPartialAnnotations. I think this should be not hard to fix (with some dedicated cleaner support). I can do this in a follow-up PR soon.
• Around a dozen incremental tests are skipped in parallel mode because order of error messages is more unstable now (which is expected). To be clear, we still group errors per module, but order of modules is much less predictable now. If there are no objections, I am going to ignore order of modules when comparing errors in incremental tests in a follow-up PR.
• When inferred type variable variance is not ready, we fall back to covariance, see testPEP695InferVarianceNotReadyWhenNeeded. However, when processing function/method bodies in a later phase, variance is ready more often. Although this is an improvement, it creates an inconsistency between parallel mode, and regular mode. I propose to address this by making the two-phase logic default even without parallel checking, see below.
• Finally, there are few edge cases with --local-partial-types when behavior is different in parallel mode, see e.g. testLocalPartialTypesWithGlobalInitializedToNone. Again the new behavior is IMO clearly better. However, it again creates an inconsistency with non-parallel mode. I propose to address this by enabling two-phase (interface then implementation) checking whenever --local-partial-types is enabled (globally, not per-file), even without parallel checking. Since --local-partial-types will be default behavior soon (and hopefully the only behavior at some point), this will allow us to avoid discrepancies between parallel and regular checking. @JukkaL what do you think?

[ilevkivskyi]

Oh btw, @JukkaL I think there is a bug in misc/diff-cache.py that may cause spurious diffs, see a TODO I added.

[ilevkivskyi]

All things in (small) mypy_primer are either good or neutral.

[hauntsaninja]

Could be worth adding a test for the discord.py improvement

[JukkaL]

I'm planning to test this on a big internal repo (probably tomorrow). I'll also try parallel checking again -- last time memory usage was too high to use many workers, but things should be better now.

[JukkaL]

I'm seeing mypy parallel run crashes with this PR when type checking the biggest internal codebase at work, but I'm not sure if they are caused by this -- this may just change the order of processing so that a pre-existing issue gets triggered. I will continue the investigation after the long weekend.

[ilevkivskyi]

@JukkaL can you post a traceback (and maybe a snippet of code where the crash happens)? It may well be some implicit assumption breaks when type-checking functions after top-levels.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "Literal[True]")  [assignment]
+ discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "T")  [assignment]
- discord/interactions.py:1109: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1255: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1645: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/webhook/async_.py:969: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]

cki-lib (https://gitlab.com/cki-project/cki-lib)
- cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" in typed context  [no-untyped-call]
+ cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" of "RefreshKerberosTicket" in typed context  [no-untyped-call]

[JukkaL]

The internal codebase generates some syntax errors because of an issue with the native parser. After working around the syntax errors, the parallel run completes, so the crashes may be related to syntax errors. However, there are a handful of false positives. Also, this regresses performance -- now parallel checking with two workers is slower than sequential checking (about 10% slower), on macOS. On master parallel checking with two workers is about 13% faster (which is still not great).

When looking at top output while a parallel run is active, each worker process is at only about 80% to 90% CPU utilization. It's possible that the added communication/synchronization overhead slows things down, at least on macOS. I'll measure performance on Linux next. I will also try to reproduce the crashes and provide tracebacks.

[ilevkivskyi]

Also, this regresses performance -- now parallel checking with two workers is slower than sequential checking (about 10% slower), on macOS. On master parallel checking with two workers is about 13% faster (which is still not great).

TBH this is really weird. Can you try running with --dump-build-stats to understand why it is so slow? Even the master number is worse than what you mentioned before in the very first PR #20280 (comment), while self-check performance improved ~50% since then.

[JukkaL]

I used 2 workers above instead of 3 in my older comment. I can try using 3 workers as well, I think I should have enough RAM for it.

[ilevkivskyi]

Also you can check communication overhead using --num-workers=0 (in-process checking) vs --num-workers=1 (checking with one separate worker).