PyInterpreterState.config.int_max_str_digits Should Not Be Modified

@gpshead

PyConfig is for configuring the runtime during initialization, not for storing runtime state. Currently in sys.set_int_max_str_digits() we are modifying the interpreter's PyConfig directly.

That field should never be modified outside runtime init. Instead, the config value should be copied to a field on PyInterpreterState and that is what should be modified by sys (and used in longobject.c).

In fact, this is mostly what we were doing until a couple weeks ago (and what we still are doing in 3.11 and earlier). This was changed on main (3.12) in gh-96944. The fix is relatively trivial.

CC @gpshead

Linked PRs

gh-98417: Stop modifying PyConfig in sys.set_int_max_str_digits() #152869

I'm pretty sure I had a version of the code doing exactly this originally and was guided towards making it simpler by not duplicating the field between interp->config and interp itself. The resolved conversation #96944 (comment) alludes to that at least.

Do we document the semantics of PyConfig being read-only anywhere?

@gpshead

@gpshead

Do we document the semantics of PyConfig being read-only anywhere?

I agree about documenting the read-only-after-init constraint somewhere. I was thinking of a comment right above the PyConfig declaration (in initconfig.h). However, you just ran into the mutability situation. What would have been the best way to alert you to the constraint?

@zooba

To be clear, I'm operating under the assumption that the cached PyConfig copy should be treated as read-only after init. That is what PEP 432 implies, it's what makes sense to me (config is config and state is state), and @zooba agreed in a private conversation. From #96944 (comment), it sounds like that might not be a universal opinion though.

FWIW, PEP 587 doesn't say anything about read-only-after-init, but PEP 432 (from which 587 was partially derived) says:

These are snapshots of the initial configuration settings. They are not modified by the interpreter during runtime

The C-API docs don't say anything about config mutability-after-init. That may be fine though, since the fact that we cache a copy of the original config on each interpreter state is an internal implementation detail.

The resolved conversation #96944 (comment) alludes to that at least.

That comment is more about preserving the original config value vs. updating during init, isn't it? The question of the source of the information during runtime (and where mutable state lives) seems like a separate one.

Updating the PyConfig struct definition comment to say it should be treated read-only after init makes sense to me.

I'd have seen that when making changes. I'm much less likely to go back and read PEPs as they're more often initial design ideas rather than policies and actual implementation.

gh-98417: Store int_max_str_digits on the Interpreter State (GH-98418)

What are advantages of having a read-only PyConfig? For me, it's fine to modify it. But if its value is copied somewhere (ex: sys module), it's good to attempt to keep both consistent (whenever possible). Currently, _testinternalcapi.set_config() always copies PyConfig.module_search_paths to sys.path, erasing sys.path change done after early Python init, which makes the function annoying in practice and that's why I don't want to expose it in public.

My main motivation to design PyConfig was to put all inputs at the same place. My ideal would be that PyConfig would be the only input to initialize Python.

But once Python is initialized, for me, it's convenient to reuse this structure for a few variables. The only case I'm aware where the "initial" value matters is sys.orig_argv that I added recently (Python 3.10), since Python makes changes to build sys.argv. Duplicating values between PyConfig and PyInterpreterState can lead to inconsistencies and wasting memory, no?

We should've (and still can) allow PyConfig to be fully provided by the caller, rather than making them go via our memory management functions. This would be very convenient for some situations, but relies on us not modifying the structure at all.

Honestly, we shouldn't even copy it around the way we do. It should exist once, and if we need to refer back to it we can get to it through a pointer (bearing in mind that the host app owns it, not us, and so they might have modified it themselves). But really, it just looks like a set of parameters to Py_Initialize (and friends), rather than a runtime structure. If it doesn't survive beyond the initialize call, we should be able to handle that.

Merge branch 'main' into typewatch

Currently, PyConfig sits between two chairs: it's used to initialized Python (as expected), but it also stays during the whole Python lifecycle.

I hesitated to build a separated structure to store this configuration in PyInterpreterState: it would avoid storing things which become useless once Python is initialized. The problem is that it caused a lot of code to be duplicated, since we would have two similar but different structures. I chose to put PyConfig simply to keep the code as simple as possible. So yeah, we waste a few bytes in memory, but a single structure has to be handled in the C code.

Keeping PyConfig around also makes Py_NewInterpreter() simpler: it just copies the PyConfig from the current interpreter. By the way, _Py_NewInterpreterFromConfig() allows to run an interpreter with a different configuration.

Honestly, we shouldn't even copy it around the way we do.

Py_InitializeFromConfig() is a wild beast: it changes the memory allocator with implicit Python pre-initialization. If we do everything in a single PyConfig, the ownership of memory allocated on the stack is not well defined, and it's also unclear which memory allocator should be used to release the memory.

Moreover, PyConfig design is to collect all data to initialize Python without modifying Python, and then "write" the configuration: _PyConfig_Write(). This design also allows to override/replace PyConfig at runtime, currently implemented as _testinternalcapi.set_config(). This feature is also used by Modules/getpath.py if I followed correctly.

Yeah, PyConfig API is weird and surprising, but they are reasons for its special design :-)

Keeping it around is great! I love it! Modifying it after we've initialised is the issue. It needs to be treated as read-only, because it's not our memory.

And really, getpath shouldn't modify it either, but should go directly to setting up sys itself. We still keep the config around, but would need to re-run getpath if we're going to base new interpreters off the original config rather than the current setup of the interpreter creating them. However, we're now committed to modifying the original config for users (well, for our own tests), and I didn't want to block the getpath rewrite on a deprecation cycle.

If we're still treating config variables as live sources of truth, then this issue is still relevant.

If there's a value that can change at runtime (not just because the user/creator set it), then it should be directly in the interpreter state, not in the struct used to create it.

PyConfig_Set was added to apply settings into a struct that may change over releases (as you well know, since it's your API 😉 ). It was never intended for changing runtime settings at runtime, because that's not what the config structs are for.

This issue can be closed when the field for the runtime-modifiable setting is moved to the interpreter state directly, and that location is the one that gets updated by the sys functions.

pythongh-98417: Stop modifying PyConfig in sys.set_int_max_str_digits()

pythongh-98417: Drop the added PyConfig comment in initconfig.h

pythongh-98417: Tighten the NEWS entry wording

pythongh-98417: Stop modifying PyConfig in sys.set_int_max_str_digits()

PR #152869 basically amounts to a one line change plus a unittest for this. do we want it? i do not consider this a bugfix as it is an observable behavior change so i wouldn't backport it past 3.15.

As I wrote previously, PyConfig_Set() now modifies directly PyInterpreterState.config. Example setting PyInterpreterState.config.cpu_count (int):

$ python3.16
>>> import _testcapi, os
>>> import _testcapi, os
>>> os.cpu_count()
12

>>> _testcapi.config_set("cpu_count", 123)
>>> os.cpu_count()
123

PyInterpreterState.config is no longer read-only. But I don't consider that it's a configuration thing, it's just part of the interpreter state. The fact that it uses PyConfig is just an implementation detail.

I understand that some people are unhappy about that since PyConfig was designed first to configure the Python initialization. IMO it's ok to reuse PyConfig in PyInterpreterState, because it's just convenient.

If you consider that it's a design issue, someone should step in and propose a patch to remove PyInterpreterState.config: move variables to PyInterpreterState directly.

Until we actually remove PyInterpreterState.config, I would prefer to keep PyInterpreterState.config consistent, even if the value is stored elsewhere. For example, even if we do have a PyInterpreterState.long_state.max_str_digits variable, there is still a PyInterpreterState.config.int_max_str_digits copy. IMO it's better to keep PyInterpreterState.config.int_max_str_digits updated when the value is modified by sys.set_int_max_str_digits() or PyConfig_Set("int_max_str_digits", value).

But I don't consider that it's a configuration thing, it's just part of the interpreter state.

Then your consideration is wrong.

There's value in keeping the original configuration around even as state changes, so "remove the field" is too drastic. Treating the field as a read-only copy of the configuration used to create the interpreter is the right way here.

So we consider that the design issue is exactly what this issue says, and the only pushback is that you've misunderstood the feature.

i do not consider this a bugfix as it is an observable behavior change so i wouldn't backport it past 3.15.

Bugfixes generally are, since bugs are observable behaviours that we don't want 😉 But I'm okay with not backporting further than 3.15, as long as we've got the main branch behaving properly so that we don't add more of these, and having it in 3.15 makes it a bit safer for embedders who might not expect their config structs to be changing.

Navigation Menu

Repository navigation

PyInterpreterState.config.int_max_str_digits Should Not Be Modified #98417

Linked PRs

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PyInterpreterState.config.int_max_str_digits Should Not Be Modified #98417

Description

Linked PRs

Activity

gpshead commented on Oct 18, 2022

ericsnowcurrently commented on Oct 19, 2022

ericsnowcurrently commented on Oct 19, 2022

ericsnowcurrently commented on Oct 19, 2022

gpshead commented on Oct 19, 2022

vstinner commented on Oct 19, 2022

zooba commented on Oct 20, 2022

vstinner commented on Nov 3, 2022

zooba commented on Nov 7, 2022

6 remaining items

zooba commented on Dec 9, 2025

gpshead commented on Jul 2, 2026

vstinner commented on Jul 2, 2026

zooba commented on Jul 2, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions