When using `python -bb`, `struct.calcsize` raises a warning when used with str argument after being used with bytes (might be a larger problem with dicts)

[tadeu]

BPO	41777
Nosy	@mdickinson, @meadori, @serhiy-storchaka, @The-Compiler, @tadeu

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-09-13.13:23:48.243>
labels = ['interpreter-core', '3.8', 'type-bug', 'library']
title = 'When using `python -bb`, `struct.calcsize` raises a warning when used with str argument after being used with bytes (might be a larger problem with dicts)'
updated_at = <Date 2020-09-13.14:24:57.143>
user = 'https://github.com/tadeu'

bugs.python.org fields:

activity = <Date 2020-09-13.14:24:57.143>
actor = 'The Compiler'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2020-09-13.13:23:48.243>
creator = 'tadeu'
dependencies = []
files = []
hgrepos = []
issue_num = 41777
keywords = []
message_count = 4.0
messages = ['376836', '376837', '376838', '376839']
nosy_count = 5.0
nosy_names = ['mark.dickinson', 'meador.inge', 'serhiy.storchaka', 'The Compiler', 'tadeu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41777'
versions = ['Python 3.8']

Linked PRs

• gh-85943: Fix BytesWarning in the struct format cache under -bb #153627
• [3.15] gh-85943: Fix BytesWarning in the struct format cache under -bb (GH-153627) #153834
• [3.14] gh-85943: Fix BytesWarning in the struct format cache under -bb (GH-153627) #153835
• [3.13] gh-85943: Fix BytesWarning in the struct format cache under -bb (GH-153627) #153836

[tadeu]

Here is the inconsistent behavior, when running with python -bb (or just python -b), caused by an internal cache:

    >>> import struct
    >>> struct.calcsize(b'!d')  # cache for '!d' uses bytes
    8
    >>> struct.calcsize('!d')  # so there's a warning when trying to use str
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    BytesWarning: Comparison between bytes and string
    >>> struct.calcsize('>d')  # cache for '>d' uses str
    8
    >>> struct.calcsize(b'>d')  # so now the warning is inverted, it shows up when trying to use bytes
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    BytesWarning: Comparison between bytes and string
    >>> struct.calcsize('>d')  # no problem when using str
    8

Note that this might be caused by a possible larger problem when dealing with keys of different string types in dicts under python -b (or python -bb):

    $ python
    Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> d={}
    >>> d['a']=1
    >>> d[b'a']=2
    >>> d['a']
    1
    >>> d[b'a']
    2


    $ python -bb
    Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> d={}
    >>> d['a']=1
    >>> d[b'a']=2
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>

I'm not sure if this warning is intentional, since in Python 3 there seems to be no special reason for dicts to try to compare 'a' with b'a' (other than possible implementation details).

Note: this is from an issue found here: pytest-dev/pytest-xdist#596

[The-Compiler]

Taking the freedom of adding people involved in the struct module to the nosy list.

[tadeu]

I'm not sure if this warning is intentional, since in Python 3 there seems to be no special reason for dicts to try to compare 'a' with b'a' (other than possible implementation details).

Okay, there's one special reason, it's the fact that 'a' and b'a' have the same hash. I'm not sure about the expected behavior, though.

[The-Compiler]

Ah, also see https://bugs.python.org/issue21071#msg292409 where the same thing was mentioned as part of another issue as well.

After some discussions in the Python IRC channel, I guess it's acceptable for dicts to raise a ByteWarning here - after all, there *is* a comparison between str/bytes going on here. It might be an implementation detail, but so is e.g. b'a' in ['a'] and I'd certainly expect that to give me a warning/error with -b/-bb.

So I guess if struct continues to accept bytes as format string, it should probably decode them to ASCII or something before interacting with the cache?