Skip to content

Clarify the required behaviour of locals() #62160

Closed
Closed
Clarify the required behaviour of locals()#62160
Assignees
ncoghlan
Labels
3.7 (EOL)end of lifedocsDocumentation in the Doc dirtype-bugAn unexpected behavior, bug, or error

Description

@ncoghlan
ncoghlan
opened on May 12, 2013
BPO 17960
Nosy @gvanrossum, @ncoghlan, @benjaminp, @ezio-melotti, @njsmith, @florentx, @xgid
PRs
  • [PEP 558 - WIP] bpo-30744: Trace hooks no longer reset closure state #3640
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ncoghlan'
closed_at = None
created_at = <Date 2013-05-12.07:46:47.489>
labels = ['type-bug', '3.7', 'docs']
title = 'Clarify the required behaviour of locals()'
updated_at = <Date 2017-10-11.02:16:40.391>
user = 'https://github.com/ncoghlan'

    bugs.python.org fields:

    activity = <Date 2017-10-11.02:16:40.391>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2013-05-12.07:46:47.489>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 17960
keywords = ['patch']
message_count = 9.0
messages = ['188986', '188987', '296880', '296909', '297019', '297328', '301742', '302423', '304100']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'ncoghlan', 'benjamin.peterson', 'ezio.melotti', 'njs', 'flox', 'xgdomingo']
pr_nums = ['3640']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17960'
versions = ['Python 3.7']

    Activity

    1. ncoghlan commented on May 12, 2013

      @ncoghlan
      ncoghlan
      on May 12, 2013
      ContributorAuthor

      As proposed at [1], I would like to tighten up the definition of locals so that defining enum members programmatically is officially supported behaviour.

      I'll come up with a patch soonish.

      [1] http://mail.python.org/pipermail/python-dev/2013-May/125917.html

    2. ncoghlan
      self-assigned this
      on May 12, 2013
    3. ncoghlan
      added
      docsDocumentation in the Doc dir
      type-bugAn unexpected behavior, bug, or error
      on May 12, 2013

    4. ezio-melotti commented on May 12, 2013

      @ezio-melotti
      ezio-melotti
      on May 12, 2013
      Member

      See also bpo-17546 and bpo-7083.

    5. ncoghlan
      removed their assignment
      on Jun 28, 2015
    6. ncoghlan
      added
      3.7 (EOL)end of life
      on Jun 26, 2017

    7. ncoghlan commented on Jun 26, 2017

      @ncoghlan
      ncoghlan
      on Jun 26, 2017
      ContributorAuthor

      Since "soonish" turned out to be "4 years and counting", copying in the specifics of the proposal in from the old python-dev thread:

      1. While nominally undefined, in practice lots of Python programs depend on the locals() builtin behaving exactly how it behaves in CPython.

      2. PyPy at least has replicated that behaviour faithfully (to the extent of replicating our weird trace function related misbehaviour, as recently pointed out in issue bpo-30744)

      3. For module scopes and class scopes (and the corresponding forms of exec and eval), the expected behaviour is relatively straightforward to both define and implement:

      • at module scope, as well as when using exec() or eval() with a
        single namespace, locals() must return the same thing as globals(),
        which must be the actual execution namespace. Subsequent execution may
        change the contents of the returned mapping, and changes to the
        returned mapping must change the execution environment.

      • at class scope, as well as when using exec() or eval() with separate
        global and local namespaces, locals() must return the specified local
        namespace (which may be supplied by the metaclass __prepare__ method
        in the case of classes). Subsequent execution may change the contents
        of the returned mapping, and changes to the returned mapping must
        change the execution environment. For classes, this mapping will not
        be used as the actual class namespace underlying the defined class
        (the class creation process will copy the contents to a fresh
        dictionary that is only accessible by going through the class
        machinery).

      1. For function scopes, the appropriate semantics are less clear, as what CPython currently does is fairly weird and quirky.
      • actual execution uses the fast locals array and cell references (for nonlocal variables)
      • there's a PyFrame_FastToLocals operation that populates the frame's "f_locals" attribute based on the current state of the fast locals array and any referenced cells
      • a direct reference to f_locals is returned from locals(), so if you hand out multiple concurrent references, then all those references will be to the exact same dictionary
      • the two common calls to the reverse operation, PyFrame_LocalsToFast, were removed in the migration to Python 3: exec is no longer a statement and hence can longer affect function local namespaces, and the compiler now disallows the use of "from module import *" operations at function scope
      • however, two obscure calling paths remain: PyFrame_LocalsToFast is called as part of returning from a trace function, and you can also still inject the IMPORT_STAR opcode when creating a function directly from a code object rather than via the compiler

      It would be a lot simpler to document the expected behaviour at function scope if locals() were to be updated to return a true snapshot (i.e. a copy of f_locals, rather than a direct reference), with direct access to the shared locals reference requiring going through the frame attribute. That way, trace functions could still modify local variables (since they use frame.f_locals), but setting a trace function wouldn't suddenly have the side effect of making modifications to locals() take effect at function scope.

    8. gvanrossum commented on Jun 26, 2017

      @gvanrossum
      gvanrossum
      on Jun 26, 2017
      Member

      I've tried thinking though a few scenarios, and I think I'm +1 (or at least
      +0 or +0.5) on the proposed change to locals(), and of course I'm happy
      that we're going to specify its behavior better.

    9. ncoghlan commented on Jun 27, 2017

      @ncoghlan
      ncoghlan
      on Jun 27, 2017
      ContributorAuthor

      Guido: perhaps I should run this update through the PEP process?

      Even though the actual proposed change is only to a pretty obscure edge case (having multiple concurrent live references to the result of locals() for a function namespace), the extra visibility should help developers of alternative implementations be clear on what is happening.

    10. gvanrossum commented on Jun 29, 2017

      @gvanrossum
      gvanrossum
      on Jun 29, 2017
      Member

      Yeah, I think a short PEP would be helpful here.

    11. ncoghlan
      self-assigned this
      on Jun 30, 2017

    12. ncoghlan commented on Sep 8, 2017

      @ncoghlan
      ncoghlan
      on Sep 8, 2017
      ContributorAuthor

      Posted as PEP-558:

    13. ncoghlan commented on Sep 18, 2017

      @ncoghlan
      ncoghlan
      on Sep 18, 2017
      ContributorAuthor

      Status update: I've posted an initial PR to bpo-30744 that relies on the trace hook semantic change proposed in the PEP to resolve the trace hook/cell reference incompatibility reported there.

      That provides confidence that it really is only the semantics of *trace hooks* that we need to change, rather than anything about locals() or frame.f_locals in general.

      So the next steps will be to do a final editing pass on the current PEP to account for the reference implementation, and then send it to python-dev for official review and pronouncement.

    14. ncoghlan commented on Oct 11, 2017

      @ncoghlan
      ncoghlan
      on Oct 11, 2017
      ContributorAuthor

      Nathaniel raised a valid concern about the draft PEP over in https://bugs.python.org/issue30744#msg302475, so I've been considering whether or not it would be possible to make the write-through proxy idea work without introducing other problems.

      I think I have a workable design concept for that approach now: https://bugs.python.org/issue30744#msg304099

      So the next step will be to see if that actually does work as well as I think it will, and if so, update the PEP accordingly.

    15. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    16. ncoghlan commented on May 6, 2024

      @ncoghlan
      ncoghlan
      on May 6, 2024
      ContributorAuthor

      With PEP 667 implemented for 3.13b1 (#115153) and the locals() docs updated accordingly (#118265), this can finally be marked as resolved!

    17. ncoghlan
      closed this as completedon May 6, 2024

    18. gvanrossum commented on May 6, 2024

      @gvanrossum
      gvanrossum
      on May 6, 2024
      Member

      Hm, there's more docs work to be done though. The new C APIs mentioned in the PEP haven't been added there and the deprecated ones haven't been marked as deprecated. The descriptions of the deprecated ones may also have to be updated to explain what they actually do (not much, in most cases). Never mind, see gh-74929.

    19. ncoghlan commented on May 10, 2024

      @ncoghlan
      ncoghlan
      on May 10, 2024
      ContributorAuthor

      Yeah, this part is done (since it's just about the Python level API of locals()), but there are other related open issues that can't be considered complete yet.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.7 (EOL)end of lifedocsDocumentation in the Doc dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions