Repository navigation
the usage of threshold1 for the GC is not correctly explained #144960
Description
Activity
I'm not sure we have more to add than what you already cited.
There is a constant for this purpose:
Lines 1363 to 1365 in 1ddb412
If ever you face performance issues and want to spot whether the gc could be the culprit, it is useful to know that this SCAN_RATE_DIVISOR is 10 and therefore, the incremental gc on the old generation will work by bunch of 1/(threshold1 * SCAN_RATE_DIVISOR). The fact that threshold1 default to 10 and the documentation says the collected old generation is inversely proportional to threshold1, so 1% with threshold 10 (without any additional information) makes the reader wonder whether there is a typo.
From my perspective it is not clear why you use SCAN_RATE_DIVISOR instead of just passing threshold1=100 and compute 1/threshold1. I do not challenge this choice (I didn't even looked at the source code), I am just spotting possible user misunderstanding of the usage of threshold1 that could be fixed by saying that the computation is 1/(threshold1*10)
From my perspective it is not clear why you use SCAN_RATE_DIVISOR instead of just passing threshold1=100 and compute 1/threshold1.
It is done for "compatibility", let me quote myself from another issue:
Before 3.14 we have default values for thresholds like (2000, 10, 10). This means that after 100 gc runs (young + middle) we run full collection.
For incremental gc we changed meaning of the second threshold value, and this change documented:The fraction of the old generation that is collected is inversely proportional to threshold1. The larger threshold1 is, the slower objects in the old generation are collected. For the default value of 10, 1% of the old generation is scanned during each collection.
This means that for default values (2000, 10, 0) after 100 runs we do collection for whole heap like before 3.14.
If you have any problems with GC's performance, feel free to open such issues. On the other hand, it may be useful to build CPython with
Py_STATS(--enable-pystats), and look at it. Moreover, I found useful to instrument a GC and build graphs like this - #142001 (comment).Unfortunately, the
work_to_dovalue, which we control via threshold1, does not always work as you (and I) may expect.Now that the generational GC has landed, I'll close this as obsolete.
Documentation
In https://docs.python.org/3/library/gc.html#gc.set_threshold it is explained that "The fraction of the old generation that is collected is inversely proportional to threshold1." As the default value of threshold1 is 10, we expect the fraction to be 1/10=0.1, which is 10% and not 1%.
No additional information is provided in https://github.com/python/cpython/blob/3.14/InternalDocs/garbage_collector.md
May be there is a constant factor used to compute the inversely proportional function, in that case it would be useful to disclose it in the documentation.
Linked PRs