Implicit type aliases not recognised with PEP 604 + import cycles

[AlexWaygood]

This bug was originally surfaced by python/typeshed#6612. It is hard to tell if it is the only bug that was revealed by that PR, or if there are other PEP 604-related bugs in mypy.

---

Top-line summary

If a single line in typeshed/stdlib/_typeshed/__init__.pyi is changed from using Union to PEP 604-style syntax, a large number of errors are output by mypy:

--- a/stdlib/_typeshed/__init__.pyi
+++ b/stdlib/_typeshed/__init__.pyi
@@ -189,7 +189,7 @@ ReadOnlyBuffer = bytes  # stable
 # for it. Instead we have to list the most common stdlib buffer classes in a Union.
 WriteableBuffer = Union[bytearray, memoryview, array.array[Any], mmap.mmap, ctypes._CData]  # stable
 # Same as _WriteableBuffer, but also includes read-only buffer types (like bytes).
-ReadableBuffer = Union[ReadOnlyBuffer, WriteableBuffer]  # stable
+ReadableBuffer = ReadOnlyBuffer | WriteableBuffer  # stable

Mypy errors

stdlib\io.pyi:44: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:44: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:54: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:54: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:61: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:61: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:74: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:74: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:104: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:104: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:52: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:52: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:53: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:53: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:55: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:55: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:64: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:64: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ctypes\__init__.pyi:89: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ctypes\__init__.pyi:89: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\builtins.pyi:672: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\builtins.pyi:672: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:9: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:9: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:10: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:10: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:11: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:11: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:23: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:24: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:25: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:25: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\lzma.pyi:90: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\lzma.pyi:90: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:19: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:19: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:30: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:30: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:31: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:31: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:37: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:37: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:42: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:42: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:12: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:12: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:16: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:16: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:19: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:19: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:20: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:20: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:21: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:21: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:22: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:22: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:23: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:24: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:25: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:25: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:49: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:49: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:56: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:56: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:60: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:60: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:70: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:70: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:72: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:72: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:89: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:89: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:92: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:92: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:93: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:93: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:94: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:94: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:14: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:14: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:560: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:560: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:561: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:561: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:563: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:563: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:565: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:565: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\socket.pyi:603: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\socket.pyi:603: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\gzip.pyi:136: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\gzip.pyi:136: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\bz2.pyi:127: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\bz2.pyi:127: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\bz2.pyi:128: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\bz2.pyi:128: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:343: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:343: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:344: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:344: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:346: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:346: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:348: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:348: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

---

Investigation

This bug is hard to pin down. I have not managed to reproduce it outside of the typeshed repo.

I have managed to reproduce the bug in a branch of typeshed in which the vast majority of the stdlib has been deleted. This indicates to me that the bug has something to do with an interaction between builtins, _typeshed and mmap. In this "slimmed-down" version of typeshed, I see the following errors from mypy:

stdlib\builtins.pyi:677: error: Name "ReadableBuffer" is not defined
stdlib\io.pyi:5: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

---

Notes on the bug

The bug can be reproduced if ReadableBuffer is defined like this in _typeshed:

import mmap
WriteableBuffer = mmap.mmap
ReadableBuffer = bytes | WriteableBuffer

However...

(1) It cannot be reproduced if ReadableBuffer is defined like this in typeshed:

import mmap
ReadableBuffer = bytes | mmap.mmap

(2) Nor can it be reproduced if ReadableBuffer is defined like this in typeshed:

from mmap import mmap
WriteableBuffer = mmap
ReadableBuffer = bytes | WriteableBuffer

(3) If the definition of ReadableBuffer is moved to builtins, the errors in builtins and io that you see in "slimmed-down typeshed" continue to be reported by mypy.

(4) The errors in the io module that you see in "slimmed-down typeshed" do not exist if io defines its own version of ReadableBuffer, instead of importing the definition from _typeshed or builtins.

---

Expected behaviour

No error should be reported; the semantics of typing.Union and PEP 604 union-type expressions should be identical.

---

mypy version

0.930. Reproduced on many Python versions by the typeshed CI; local testing done on Python 3.10.

---

To reproduce

1. Either clone the typeshed master branch, and apply the above diff manually, or clone my "slimmed-down typeshed branch".
2. cd into your clone of the typeshed repo.
3. Run mypy stdlib/abc.pyi --custom-typeshed-dir . (the precise file you run mypy on is irrelevant).

---

Cc. @sobolevn, @JukkaL, @Akuli

[hauntsaninja]

I think you miiiiight be able to use PEP 613 as a workaround.
Also note, your slimmed down branch is a little broken, ReadableBuffer does seem actually undefined.

[hauntsaninja]

Import cycles involving builtins can do some scary things, here's a bug I ran into: #11535

[hauntsaninja]

I did some tracing of the code involved.

Looking at semanal's can_be_type_alias (called by check_and_set_up_type_alias, called by visit_assignment_stmt).

mypy/mypy/semanal.py

Line 2190 in 1beed9a

def can_be_type_alias(self, rv: Expression, allow_none: bool = False) -> bool:

If we encounter ReadableBuffer = Union[...], we have rv is a IndexExpr, we go into is_type_ref and typing.Union is included in valid_refs = type_constructors so we return True and all is well. We don't even look at the contents of the Union.

If we encounter ReadableBuffer = ReadOnlyBuffer | WriteableBuffer we check to see if both sides of "|" can be a type alias.

This check succeeds for the left side ('NameExpr(ReadOnlyBuffer [_typeshed.ReadOnlyBuffer])'), but fails for the right side 'NameExpr(WriteableBuffer)'.

In : self.lookup(rv.left.name, rv).node
<mypy.nodes.TypeAlias object at 0x106f8c2c0>
In : self.lookup(rv.right.name, rv).node
<mypy.nodes.PlaceholderNode object at 0x106f87e20>

So mypy knows that ReadOnlyBuffer is a type alias and so is_type_ref is okay with it. But all it has for WriteableBuffer is a placeholder (likely because of the import cycles), and it is not okay with that -- if n and isinstance(n.node, PlaceholderNode) and n.node.becomes_typeinfo is False.

So two things that could mechanically fix this:

1. If WriteableBuffer wasn't a placeholder. It's WriteableBuffer = Union[...], so mypy could easily know at this point that it is a type alias, even if ... involves horrendous import cycles. Although obviously we'd want our solution to still work if WriteableBuffer used PEP 604.
2. It's true that WriteableBuffer is indeed not going to become a type info, but it is going to become a type alias and that should count for the purposes of is_type_ref. It looks like the placeholder's becomes_typeinfo is basically just used for this kind of thing, so I think we could repurpose it as becomes_type.

Unfortunately, I'm not yet sure what the best way to ensure either of those.

The secret to both lies back in visit_assignment_stmt:

mypy/mypy/semanal.py

Line 2070 in 1beed9a

if self.found_incomplete_ref(tag) or self.should_wait_rhs(s.rvalue):

I don't fully understand the reference tagging thing, but tldr the import cycle causes us to skip calling check_and_set_up_type_alias for WriteableBuffer for now, so all mypy has is a placeholder node.

Moreover, the placeholder's becomes_typeinfo is False, because we call mark_incomplete without specifying becomes_typeinfo=True

mypy/mypy/semanal.py

Line 2075 in 1beed9a

self.mark_incomplete(expr.name, expr)

I'm not sure how we could know becomes_typeinfo=True at this point, although we could know could_become_type.

.... 🤷
So next step is probably to see the impact of changing the semantics of becomes_typeinfo to could_become_type. Unless you see a better fix.

Anyway, for typeshed purposes, maybe just use PEP 613 :-) (works because we don't need the result of can_be_type_alias)

[AlexWaygood]

Also note, your slimmed down branch is a little broken, ReadableBuffer does seem actually undefined.

It's defined here in my branch, no? 🙂 note that my branch doesn't make any kind of sense from a typing perspective, I was just trying to delete as many modules as possible while still being able to reproduce the bug!!

[AlexWaygood]

I did some tracing of the code involved.

Thanks for taking the time to look into it!! This is a great rundown.

Anyway, for typeshed purposes, maybe just use PEP 613 :-) (works because we don't need the result of can_be_type_alias)

Unfortunately, I think PEP 613 may still be unuseable for typeshed purposes — I'm not sure if pytype supports it yet 😕 (google/pytype#787) but, worth a shot!

For this specific instance of the bug, it seems like it can be fixed just by doing from mmap import mmap in _typeshed (but my worry is that quite a few things are imported in typeshed/stdlib/builtins.pyi, so there might be other instances of this bug — haven't investigated that yet...)

[AlexWaygood]

Also note, your slimmed down branch is a little broken, ReadableBuffer does seem actually undefined.

Oh wait, yes, lol — I was experimenting to see if the io errors are still there if it is imported in io from _typeshed, but isn't imported in builtins from _typeshed. (Answer: yes, they are — which I think is because other things are still imported from _typeshed in builtins, meaning you still get cyclic references.)

I left that in by mistake when I pushed to my remote branch — sorry for the confusion!!

[AlexWaygood]

I left that in by mistake when I pushed to my remote branch — sorry for the confusion!!

"smol typeshed" should now be fixed by AlexWaygood/typeshed@79f1d16 -- the only error from mypy is now:

stdlib\io.pyi:5: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases