[ilevkivskyi]

Ref #22063

Fixes #8478
Fixes #10281

This makes the new logic work in presence of second pass arguments (callables). I also did a little refactoring, and added some documenting comments.

Implementation is generally straightforward. I may add more tests depending on primer and mypy-issues.

[ilevkivskyi]

New error in discord.py looks like a new false positive, but I can't find an easy repro so far.

[ilevkivskyi]

OK, I found a repro

[case testTwoSidedInferenceWithParamSpecGeneric]
from typing import Callable, Generic, TypeVar, Union
from typing_extensions import ParamSpec

P = ParamSpec("P")
T = TypeVar("T")
I = TypeVar("I", bound=Union[int, str])

class C(Generic[T]): ...

def thing(func: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T: ...

def test(x: C[I]) -> list[int]: ...
ci: C[int]
f: list[int] = thing(test, ci)
[builtins fixtures/paramspec.pyi]

This is however quite niche, it is related to weakness in how we calculate joins of "generic" Parameters. I will check if there is a simple fix.

[ilevkivskyi]

The change in trio is actually good IMO, I think this is ready for review. @hauntsaninja @JukkaL please take a look if you have time.

[JukkaL]

This looks good. The only issue I found is that nested lambdas are slower to type check. Example where this matters:

from typing import Callable, Iterable, Sequence, TypeVar

T = TypeVar("T")
S = TypeVar("S")


def mapl(fn: Callable[[T], S], x: Iterable[T]) -> list[S]:
    return [fn(i) for i in x]


# The outer type context isn't needed to trigger the slowdown (the inner calls get
# context from the enclosing lambda), but it makes the first call two-sided too.
x: Sequence[object] = mapl(
    lambda v1: mapl(
        lambda v2: mapl(
            lambda v3: mapl(
                lambda v4: mapl(
                    lambda v5: mapl(
                        lambda v6: mapl(
                            lambda v7: mapl(lambda v8: mapl(lambda v9: v9, [1]), [1]),
                            [1],
                        ),
                        [1],
                    ),
                    [1],
                ),
                [1],
            ),
            [1],
        ),
        [1],
    ),
    [1],
)

It gets worse as the nesting depth increases, but this is probably not a major issue, as deeply nested lambdas are rare (though I've seen them in some real-world test code).

[JukkaL]

Typo: cntext

[JukkaL]

Typo: out solver

[JukkaL]

Typo: constrains

[ilevkivskyi]

@JukkaL Yeah, lambdas are a known problem, because we can't use expression cache inside them. The new inference scheme heavily relies on the expression cache (since its complexity is exponential without the cache). At some point we may need to think about this. One possible idea may be to store a separate stack of contexts for lambdas, and use combined context as a cache key. This will probably fix performance at least in some cases (like deeply nested calls within a single lambda), I didn't think a lot about it however.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

egglog-python (https://github.com/egraphs-good/egglog-python)
+ python/egglog/exp/array_api.py:69: error: Unused "type: ignore" comment  [unused-ignore]
+ python/egglog/exp/array_api.py:70: error: Unused "type: ignore" comment  [unused-ignore]

core (https://github.com/home-assistant/core)
- homeassistant/components/minecraft_server/api.py:95: error: Item "None" of "Any | None" has no attribute "timeout"  [union-attr]
+ homeassistant/components/fastdotcom/coordinator.py:34: error: Returning Any from function declared to return "dict[str, float] | None"  [no-any-return]

trio (https://github.com/python-trio/trio)
+ src/trio/_socket.py:1311: error: Unused "type: ignore" comment  [unused-ignore]
+ src/trio/_socket.py:1312: error: Argument 3 to "_nonblocking_helper" of "_SocketType" has incompatible type "*list[object]"; expected "Buffer"  [arg-type]
+ src/trio/_socket.py:1312: error: Argument 3 to "_nonblocking_helper" of "_SocketType" has incompatible type "*list[object]"; expected "tuple[Any, ...] | str | Buffer"  [arg-type]

pywin32 (https://github.com/mhammond/pywin32)
  ./pythonwin/pywin/test/_dbgscript.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
- version: 2.5.0+dev.b53c712057b5516f416c7f17fdf0744326d79c6c
+ version: 2.5.0+dev.9a6a30c35571d087d697a78a0fbdaded571e5a57

[ilevkivskyi]

Oh interesting, it looks like pywin32 is crashing on master. I will take a look.

[ilevkivskyi]

This looks like a regression in the new parser, it now crashes on non-UTF-8 file (containing actual latin-1 symbols) that also has a function definition in it.

@JukkaL since this is completely unrelated to this PR, I will handle it later.