(See https://github.com/python/cpython/issues/100227.) Currently the tracemalloc module has some state in `_PyRuntimeState`, including objects, which is shared by all interpreters. Interpreters should be isolated from each other, for a variety of reasons (including the possibility of a per-interpreter GIL). Isolating the module will involve moving some of the state to `PyInterpreterState` (and, under per-interpreter GIL, guarding other state with a global lock). ---- Analysis: <details> <summary>(expand)</summary> ## Allocators The module installs a custom allocator (using the PEP 445 API ([docs](https://docs.python.org/3.12/c-api/memory.html#customize-memory-allocators)). <details> <summary>(expand)</summary> <BR/> the allocator functions: | domain | func | wraps | wraps (reentrant) | actually wraps | | ---- | ---- | ---- | ---- | ---- | | <p align="center">-</p> | `tracemalloc_alloc()` | \<original `malloc()` or `calloc()`\> | \<--- | | | <p align="center">-</p> | `tracemalloc_realloc()` | \<original `realloc()` or `free()`\> | \<--- | | | <p align="center">-</p> | `tracemalloc_raw_alloc()` | `tracemalloc_alloc()` **\*** | \<original `malloc()` or `calloc()`\> | | | <p align="center">-</p> | `tracemalloc_alloc_gil()` | `tracemalloc_alloc()` | \<original `malloc()` or `calloc()`\> | | | raw | | | | | | | `tracemalloc_raw_malloc()` | `tracemalloc_alloc()` | <original `malloc()`> | `tracemalloc_raw_alloc()` | | | `tracemalloc_raw_calloc()` | `tracemalloc_alloc()` | <original `calloc()`> | `tracemalloc_raw_alloc()` | | | `tracemalloc_raw_realloc()` | `tracemalloc_realloc()` **\*** | \<original `realloc()`> | | | | `tracemalloc_free()` | <original `free()`> | \<--- | | | mem | | | | | | obj | | | | | | | `tracemalloc_malloc_gil()` | `tracemalloc_alloc_gil()` | \<--- | | | | `tracemalloc_calloc_gil()` | `tracemalloc_alloc_gil()` | \<--- | | | | `tracemalloc_realloc_gil()` | `tracemalloc_realloc()` | \<original `realloc()`\> | | | | `tracemalloc_free()` | <original `free()`> | \<--- | | **\*** Note that `tracemalloc_raw_alloc()` wraps the `tracemalloc_alloc()` call with `PyGILState_Ensure()`/`PyGILState_Release()`. Likewise for `tracemalloc_raw_realloc()` where it calls `tracemalloc_realloc()`. In no other case does an allocator function use the GILState API for any calls. </details> ## State ### Fields <details> <summary>(expand)</summary> <BR/> https://github.com/python/cpython/blob/main/Include/internal/pycore_tracemalloc.h#L57-L107 https://github.com/python/cpython/blob/main/Include/internal/pycore_runtime.h#L147 <details> <summary>raw</summary> <BR/> https://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Include/internal/pycore_tracemalloc.h#L43-L55 https://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Include/internal/pycore_tracemalloc.h#L57-L64 https://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Include/internal/pycore_tracemalloc.h#L57-L107 https://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Modules/_tracemalloc.c#L54-L55 https://github.com/python/cpython/blob/0675b8f032c69d265468b31d5cadac6a7ce4bd9c/Modules/_tracemalloc.c#L69-L76 </details> | name | type | protected by | \#ifdef | notes | | ---- | ---- | ---- | ---- | ---- | | `config` | `struct _PyTraceMalloc_Config` | | | | | . `initialized` | `enum {}` | GIL | | | | . `tracing` | `bool` | GIL | | | | . `max_nframe` | `int` | GIL | | | | `allocators` | | | | see PEP 445 ([docs](https://docs.python.org/3.12/c-api/memory.html#customize-memory-allocators)) | | . `mem` | `PyMemAllocatorEx` | GIL | | | | . `raw` | `PyMemAllocatorEx` | GIL | | | | . `obj` | `PyMemAllocatorEx` | GIL | | | | `tables_lock` | `PyThread_type_lock` | GIL | `TRACE_RAW_MALLOC` | | | `traced_memory` | `size_t` | `tables_lock` | | | | `peak_traced_memory` | `size_t` | `tables_lock` | | | | `filenames` | `_Py_hashtable_t *` | GIL | | interned; effectively a `set` of objects | | `traceback` | `struct tracemalloc_traceback *` | GIL | | a temporary buffer | | `tracebacks` | `_Py_hashtable_t *` | GIL | | interned; effectively a `set` of `traceback_t` | | `traces` | `_Py_hashtable_t *` | `tables_lock` | | void-ptr -> `trace_t` | | `domains` | `_Py_hashtable_t *` |`tables_lock` | | domain -> `_Py_hashtable_t *` (per-domain `traces`) | | `empty_traceback` | `struct tracemalloc_traceback` | ??? | | | | `reentrant_key` | `Py_tss_t` | ??? | | | notes: * each frame in `struct tracemalloc_traceback` holds a filename object * `traceback_t` is a typedef for `struct tracemalloc_traceback` * `frame_t` is a typedef for `struct tracemalloc_frame` hold objects: * `filenames` * `traceback` (filename in each frame) * `tracebacks` (filename in each frame of each traceback) * `traces` (filename in each frame of each traceback) * `domains` (filename in each frame of each traceback in each domain) </details> ### Usage <details> <summary>(expand)</summary> <BR/> simple: | name | context | get | set | | ---- | ---- | ---- | ---- | | `config` | | | | | . `initialized` | lifecycle | `tracemalloc_init()`<BR/>`tracemalloc_deinit()` | `tracemalloc_init()`<BR/>`tracemalloc_deinit()` | | . `tracing` | module | `_tracemalloc_is_tracing_impl()`<BR/>`_tracemalloc__get_traces_impl()`<BR/>`_tracemalloc_clear_traces_impl()`<BR/>`_tracemalloc_get_traceback_limit_impl()`<BR/>`_tracemalloc_get_traced_memory_impl()`<BR/>`_tracemalloc_reset_peak_impl()` | | | | C-API | `PyTraceMalloc_Track()`<BR/>`PyTraceMalloc_Untrack()`<BR/>`_PyTraceMalloc_NewReference()`<BR/>`_PyMem_DumpTraceback()` | | | | lifecycle | `tracemalloc_start()`<BR/>`tracemalloc_stop()` | `tracemalloc_start()`<BR/>`tracemalloc_stop()` | | | internal | `tracemalloc_get_traceback()` | | | . `max_nframe` | module | `_tracemalloc_get_traceback_limit_impl()` | | | | lifecycle | | `tracemalloc_start()` | | | internal | `traceback_get_frames()` | | | `allocators` | | | | | | . `mem` | lifecycle | `tracemalloc_start()` \+<BR/>`tracemalloc_stop()` | | | . `raw` | lifecycle | `tracemalloc_init()` \+<BR/>`tracemalloc_start()` \+<BR/>`tracemalloc_stop()` | | | | internal | `raw_malloc()`<BR/> `raw_free()` | | | . `obj` | lifecycle | `tracemalloc_start()` \+<BR/>`tracemalloc_stop()` | | | `tables_lock` | module | `_tracemalloc__get_traces_impl()` \+<BR/>`_tracemalloc_get_tracemalloc_memory_impl()` \+<BR/>`_tracemalloc_get_traced_memory_impl()` \+<BR/>`_tracemalloc_reset_peak_impl()` \+ | | | | C-API | `PyTraceMalloc_Track()` \+<BR/>`PyTraceMalloc_Untrack()` \+<BR/>`_PyTraceMalloc_NewReference()` \+ | | | | lifecycle | `tracemalloc_init()`<BR/>`tracemalloc_deinit()` | `tracemalloc_init()` \*<BR/>`tracemalloc_deinit()` \* | | | allocator | `tracemalloc_alloc()` \+<BR/>`tracemalloc_realloc()` \+<BR/>`tracemalloc_free()` \+<BR/>`tracemalloc_realloc_gil()` \+<BR/>`tracemalloc_raw_realloc()` \+ | | | | internal | `tracemalloc_get_traceback()` \+<BR/>`tracemalloc_clear_traces()` \+ | | | `traced_memory` | module | `_tracemalloc_get_traced_memory_impl()`<BR/>`_tracemalloc_reset_peak_impl()` | | | | internal | `tracemalloc_add_trace()` | `tracemalloc_add_trace()`<BR/>`tracemalloc_remove_trace()`<BR/>`tracemalloc_clear_traces()` | | `peak_traced_memory` | module | `_tracemalloc_get_traced_memory_impl()` | `_tracemalloc_reset_peak_impl()` | | | internal | `tracemalloc_add_trace()` | `tracemalloc_add_trace()`<BR/>`tracemalloc_clear_traces()` | | `filenames` | module | `_tracemalloc_get_tracemalloc_memory_impl()` | | | | lifecycle | | `tracemalloc_init()` \* | | | internal | `tracemalloc_get_frame()`<BR/>`tracemalloc_clear_traces()` \+ | | | `traceback` | lifecycle | `tracemalloc_stop()` | `tracemalloc_start()` \*<BR/>`tracemalloc_stop()` \* | | | internal | `traceback_new()` \+ | | | `tracebacks` | module | `_tracemalloc_get_tracemalloc_memory_impl()` | | | | lifecycle | | `tracemalloc_init()` \*<BR/>`tracemalloc_deinit()` \+ | | | internal | `traceback_new()` \+<BR/>`tracemalloc_clear_traces()` \+ | | | `traces` | module | `_tracemalloc__get_traces_impl()`<BR/>`_tracemalloc_get_tracemalloc_memory_impl()` | | | | C-API | `_PyTraceMalloc_NewReference()` | | | | lifecycle | `tracemalloc_deinit()` \+ | `tracemalloc_init()` \* | | | internal | `tracemalloc_get_traces_table()`<BR/>`tracemalloc_add_trace()` (indirect)<BR/>`tracemalloc_remove_trace()` (indirect)<BR/>`tracemalloc_get_traceback()` (indirect)<BR/>`tracemalloc_clear_traces()` \+ | | | `domains` | module | `_tracemalloc__get_traces_impl()`<BR/>`_tracemalloc_get_tracemalloc_memory_impl()` | | | | lifecycle | `tracemalloc_deinit()` \+ | `tracemalloc_init()` \* | | | internal | `tracemalloc_get_traces_table()`<BR/>`tracemalloc_remove_trace()` (indirect)<BR/>`tracemalloc_get_traceback()` (indirect)<BR/>`tracemalloc_clear_traces()` \+<BR/>`tracemalloc_add_trace()` \+ | | `empty_traceback` | lifecycle | `tracemalloc_init()` \+ | | | | internal | `traceback_new()` | | | `reentrant_key` | lifecycle | `tracemalloc_init()` \+<BR/>`tracemalloc_deinit()` \+ | | | | allocator | `tracemalloc_alloc_gil()` (indirect) \+<BR/>`tracemalloc_realloc_gil()` (indirect) \+<BR/>`tracemalloc_raw_alloc()` (indirect) \+<BR/>`tracemalloc_raw_realloc()` (indirect) \+ | | | | internal | `get_reentrant()`<BR/>`set_reentrant()` \+ | | \* the function allocates/deallocates the value (see below) \+ the function mutates the value (see below) simple (extraneous): | name | context | allocate/deallocate | get (assert-only) | | ---- | ---- | ---- | ---- | | `config.tracing` | internal | | `tracemalloc_add_trace()`<BR/>`tracemalloc_remove_trace()` | | `tables_lock` | lifecycle | `tracemalloc_init()`<BR/>`tracemalloc_deinit()` | | | `traced_memory` | internal | | `tracemalloc_remove_trace()` | | `filenames` | lifecycle | `tracemalloc_init()` | | | `traceback` | lifecycle | `tracemalloc_start()`<BR/>`tracemalloc_stop()` | `tracemalloc_start()` | | `tracebacks` | lifecycle | `tracemalloc_init()` | | | `traces` | lifecycle | `tracemalloc_init()` | | | `domains` | lifecycle | `tracemalloc_init()` | | mutation of complex fields: | name | context | initialize | finalize | clear | modify | | ---- | ---- | ---- | ---- | ---- | ---- | | `allocators.mem` | lifecycle | `tracemalloc_start()` | | | | | `allocators.raw` | lifecycle | `tracemalloc_init()`<BR/>`tracemalloc_start()` | | | | | `allocators.obj` | lifecycle | `tracemalloc_start()` | | | | | `tables_lock` | module | | | | `_tracemalloc__get_traces_impl()`<BR/>`_tracemalloc_get_tracemalloc_memory_impl()`<BR/>`_tracemalloc_get_traced_memory_impl()`<BR/>`_tracemalloc_reset_peak_impl()` | | | C-API | | | | `PyTraceMalloc_Track()`<BR/>`PyTraceMalloc_Untrack()`<BR/>`_PyTraceMalloc_NewReference()` | | | allocator | `tracemalloc_alloc()` | | | `tracemalloc_alloc()`<BR/>`tracemalloc_realloc()`<BR/>`tracemalloc_free()`<BR/>`tracemalloc_realloc_gil()`<BR/>`tracemalloc_raw_realloc()` | | | internal | | | | `tracemalloc_clear_traces()`<BR/>`tracemalloc_get_traceback()` | | `filenames` | internal | | | `tracemalloc_clear_traces()` | `tracemalloc_get_frame()`<BR/>`tracemalloc_clear_traces()` | | | lifecycle | | `tracemalloc_deinit()` | | | | `traceback` | internal | `traceback_new()` | | | | | `tracebacks` | lifecycle | | `tracemalloc_deinit()` | | | | | internal | | | `tracemalloc_clear_traces()` | `traceback_new()` | | `traces` | lifecycle | | `tracemalloc_deinit()` | | | | | internal | | | `tracemalloc_clear_traces()` | | | `domains` | lifecycle| | `tracemalloc_deinit()` | | | | | internal | | | `tracemalloc_clear_traces()` | `tracemalloc_add_trace()` | | `reentrant_key` | lifecycle | `tracemalloc_init()` | `tracemalloc_deinit()` | | | | | internal | | | | `set_reentrant()` | indirection: | name | context | direct | indirect | | ---- | ---- | ---- | ---- | | `allocators.raw` | lifecycle | `tracemalloc_start()`<BR/>`tracemalloc_copy_trace()` | `raw_malloc()` | | | | `tracemalloc_stop()` | `raw_free()` | | | internal | `traceback_new()`<BR/>`tracemalloc_add_trace()`<BR/>`tracemalloc_copy_trace()` | `raw_malloc()` | | | | `traceback_new()`<BR/>`tracemalloc_add_trace()`<BR/>`tracemalloc_remove_trace()` | `raw_free()` | | `traces` | internal | `tracemalloc_add_trace()`<BR/>`tracemalloc_remove_trace()`<BR/>`tracemalloc_get_traceback()` | `tracemalloc_get_traces_table()` | | `domains` | internal | `tracemalloc_add_trace()`<BR/>`tracemalloc_remove_trace()`<BR/>`tracemalloc_get_traceback()` | `tracemalloc_get_traces_table()` | | `reentrant_key` | allocator | `tracemalloc_alloc_gil()`<BR/>`tracemalloc_realloc_gil()`<BR/>`tracemalloc_raw_alloc()`<BR/>`tracemalloc_raw_realloc()` | `get_reentrant()` | | | | `tracemalloc_alloc_gil()`<BR/>`tracemalloc_realloc_gil()`<BR/>`tracemalloc_raw_alloc()`<BR/>`tracemalloc_raw_realloc()` | `set_reentrant()` | </details> </details> <!-- gh-linked-prs --> ### Linked PRs * gh-104508 <!-- /gh-linked-prs -->