Repository navigation
Closed
Labels
3.13only security fixesonly security fixesperformancePerformance or resource usagePerformance or resource usagetopic-typingtype-featureA feature request or enhancementA feature request or enhancement
Description
Feature or enhancement
Proposal:
As noted in https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173, typing isn't the slowest stdlib module in terms of import time, but neither is it one of the quickest. We should speed it up, if possible.
Links to previous discussion of this feature:
Linked PRs
- gh-109653:
typing.py: improve import time by creating soft-deprecated members on demand #109651 - gh-109653: Improve
enumimport time by avoiding import offunctools#109789 - gh-109653: Remove unused imports in the
Lib/directory #109803 - gh-109653: Avoid a top-level import of
typesinfunctools#109804 - gh-109653: Just import
recursive_reprindataclasses#109822 - gh-109653: Improve the import time of
email.utils#109824 - GH-109653: Defer import of
importlib.metadata._adapters#109829 - gh-109653: Fix py312 regression in the import time of
random#110221 - [3.12] gh-109653: Fix py312 regression in the import time of
random(GH-110221) #110247 - gh-109653: Defer importing
warningsin several modules #110286 - gh-109653: Improve import time of
loggingby lazy loadingtraceback#112995 - gh-109653: Speedup import of threading module #114509
- gh-109653: Improve import time of importlib.metadata / email.utils #114664
- gh-109653: Reduce import overhead of uuid module on Linux #115160
- gh-109653: What's new: Note improved import times for several stdlib modules in Python 3.13 #118697
- gh-154675: Improve import performance of
tkinterlibrary with lazy imports #148409
Reacted by Daniel Hollas, Ofek Lev, Erlend E. Aasland, Hugo van Kemenade, Pathompong Kwangtong and Brett Holman
Reacted by Ofek Lev, Hugo van Kemenade, Pathompong Kwangtong and EDM115
Reacted by Daniel Hollas, Ofek Lev, Erlend E. Aasland, Hugo van Kemenade and Pathompong Kwangtong
Metadata
Metadata
Assignees
Labels
3.13only security fixesonly security fixesperformancePerformance or resource usagePerformance or resource usagetopic-typingtype-featureA feature request or enhancementA feature request or enhancement
Activity
gh-109653: Remove unused imports in the `Lib/` directory (#109803)
AlexWaygood commentedon Sep 24, 2023
@sobolevn, the situation with
dataclassesandreprlibyou point out in #109653 (comment) is indeed interesting. From my local experiments, it does seem like the import time ofdataclassesis unaffected ifdataclassesis altered so that it just importsrecursive_reprfromreprlibrather than reimplementing that logic. The import time forfunctoolsalso doesn't really change if it vendors thereprlibcode instead of importing it; it seems like that import is really cheap.So, should we just go for cleaner code in
dataclasses, and importrecursive_reprfromreprlib? Maybe, but there are other reasons than just import time to avoid unnecessary dependencies between different stdlib modules. I think it's not the biggest issue in the world; we can probably leave the code as it is. I'll defer to thedataclassesexperts on whether or not they feel like changing this.gh-109653: Avoid a top-level import of `types` in `functools` (#109804)
pythongh-109653: Just import `recursive_repr` in `dataclasses`
hauntsaninja commentedon Sep 25, 2023
hugovk commentedon Sep 25, 2023
AlexWaygood commentedon Sep 25, 2023
ewdurbin commentedon Sep 25, 2023
hugovk commentedon Sep 25, 2023
AA-Turner commentedon Sep 25, 2023
Looking at
importlib.metadatathere's a fairly easy ~15% gain by defering the import of_adapters(email.message). I tried removing all of the annotations & imports oftyping, but this is made difficult asTraversableis a core part of the API, and it's a runtime checkable typing protocol rather than a more 'normal' (for the stdlib) ABC. The improvement over the first optimisation is therefore only 7% (20% total).A
pythongh-109653: `typing.py`: improve import time by creating soft-de…
pythongh-109653: Improve `enum` import time by avoiding import of `fu…
pythongh-109653: Remove unused imports in the `Lib/` directory (pytho…
pythongh-109653: Avoid a top-level import of `types` in `functools` (p…
DavidCEllis commentedon Sep 29, 2023
One thing that does mildly bug me about the
_recursive_reprdefinition in dataclasses is that it is modified to usefunctools.wrapsinstead of manually fixing the attributes like thereprlibimplementation. The definition of functools.wraps usespartialwhich in turn usesrecursive_reprfromreprlibfor its own__repr__1. So it's kind of usingrecursive_reprindirectly to avoid depending onrecursive_reprdirectly.dataclassesimport time is unaffected by the change here as most of the significant direct imports end up being imported in-turn byinspect. (eg:recould be delayed until needed to identify a string annotation, but that doesn't matter whileinspectis being used as the core method to gather annotations asinspectwill eagerly importre).I wrote about removing or delaying some of the imports dataclasses uses late last year but didn't end up publishing or pursuing it as #97800 made
inspect.get_annotationsthe mechanism by which dataclasses obtains the annotations and as such much more of a core part of the functionality. I figured it would be too much churn at that point and unlikely to get in. I've made this public as a gist if it is of any interest but it's a bit tangential and far too long to include here.To summarise, taken from the
-Ximporttimeresults I had at the time:Footnotes
In the python implementation of partial at least. ↩
DavidCEllis commentedon Oct 2, 2023
I had a look at
dataclassesagain as it is now and this could still be done if the relevant parts ofinspect.get_annotationsare copied into the module asrecursive_reprcurrently is.Almost any improvements also require deferring import of
inspectuntil it's needed to generate a__doc__for a missing docstring1. It also requires preventing dataclasses from making the docstring in '-OO' mode2 or there's a chance that it would make -OO slower due to the inspect import always being triggered.This does mean that classes with no docstring/with string annotations/with the future
annotationsimport will still end up with the inspect/re imports, which could be a surprising difference in performance. I'm also not sure if it would be too disruptive.The changes would look something like this:
Code changes to dataclasses
With
-Ximporttimegoing from ~20ms for dataclasses to ~4ms../python -Ximporttime -c "import dataclasses"
Before:
After:
copycould also be deferred to the inside ofasdictandastuplewhen needed to save another chunk. I don't think the inline import will make much of a difference in performance but I'd like to test that again before proposing it.If it seems worth it I could make a PR and investigate the effect of deferring
copytoo.Edit: I realised that you can use a non-data descriptor to delay the
inspectimport in this case until__doc__is accessed rather than on creation. A little more complicated but it makes the performance behaviour more sensible.3Additional changes for non-data descriptor
Footnotes
I'd like to be able to remove that but the signature code is more involved than the get_annotations code and I don't know of a nice way to replace it without possibly changing the output. ↩
Perhaps it should already be doing this? ↩
I realised almost immediately after posting this that you could probably just use the same
_DocstringMakerinstance. ↩gh-109653: Reduce the import time of `random` by 60% (#110221)
pythongh-109653: Reduce the import time of `random` by 60% (pythonGH-…
[3.12] gh-109653: Reduce the import time of `random` by 60% (GH-110221)…
gh-109653: Defer importing `warnings` in several modules (#110286)
gh-109653: Improve the import time of `email.utils` (#109824)
danielhollas commentedon Nov 24, 2023
I recently spent a sizeable amount of time trying to speedup startup time of our CLI app, and was delighted to find this issue, thanks so much @AlexWaygood for working on this! 👏
Another chunky stdlib module is
logging. It looks like a low-hanging fruit there would be to defer importingtraceback, this seems to give around 5ms improvement. Here's how the patch would look on current master:ofek commentedon Dec 9, 2023
I would like to echo the elation of the previous commenter when I encountered this ongoing effort today. As a maintainer of various CLIs and also two personal apps that run on AWS Lambda/Google Cloud Functions, this will help me tremendously!
Thank you very, very much 🙂
AlexWaygood commentedon Dec 9, 2023
Sorry for the slow response @danielhollas! That indeed looks like a pretty good idea to me. Feel free to make a PR, and ping me on the PR for review :)
You're very welcome :)
Speed up import time of logging module
Speed up import time of logging module
pythongh-109653: Improve import time of importlib.metadata / email.utils
gh-109653: Improve import time of importlib.metadata / email.utils (#…
gh-109653: Speedup import of threading module (#114509)
pythongh-109653: Improve import time of importlib.metadata / email.ut…
pythongh-109653: Speedup import of threading module (python#114509)
gh-109653: Just import `recursive_repr` in `dataclasses` (gh-109822)
gh-109653: Reduce import overhead of uuid module on Linux (#115160)
hugovk commentedon Mar 7, 2024
Great to see these improvements!
Let's add a brief overview in What's New in 3.13, at least before beta (2024-05-07).
GH-109653: Defer import of ``importlib.metadata._adapters`` (#109829)
pythongh-109653: Just import `recursive_repr` in `dataclasses` (pytho…
pythongh-109653: Reduce import overhead of uuid module on Linux (pyth…
pythonGH-109653: Defer import of ``importlib.metadata._adapters`` (py…
Squashed commit of the following:
Squashed commit of the following:
pythongh-109653: Just import `recursive_repr` in `dataclasses` (pytho…
pythongh-109653: Reduce import overhead of uuid module on Linux (pyth…
pythonGH-109653: Defer import of ``importlib.metadata._adapters`` (py…
pythongh-109653: What's new: Note improved import times for several s…
gh-109653: What's new: Note improved import times for several stdlib …
layday commentedon May 8, 2024
I believe there are futher improvements to be made here - how would you like to handle these? For example,
importlib.metadataimportszipfileat the top for a function that won't be called in the vast majority of cases. It also importsimportlib.abc, which in turn importsimportlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method inimportlib.metadatawould be harmless.AlexWaygood commentedon May 8, 2024
Sure -- I felt like the low-hanging fruit had been tackled here, and this issue hadn't seen much activity in quite a while, so I felt like it was a good time to close. But more improvements are obviously always welcome. Maybe open a new issue if you have some more targeted improvements you want to propose?
ofek commentedon May 8, 2024
Please link the new issue here so we all can start following that as well!
layday commentedon May 8, 2024
#118761
pythongh-109653: What's new: Note improved import times for several s…
pythongh-109653: `typing.py`: improve import time by creating soft-de…
pythongh-109653: Improve `enum` import time by avoiding import of `fu…
pythongh-109653: Remove unused imports in the `Lib/` directory (pytho…
pythongh-109653: Avoid a top-level import of `types` in `functools` (p…
pythongh-109653: Reduce the import time of `random` by 60% (python#11…
pythongh-109653: Defer importing `warnings` in several modules (pytho…
pythongh-109653: Improve the import time of `email.utils` (python#109824