Repository navigation
Breaking change in 7e5bae4 (importlib_metadata 5) #409
Description
Activity
Breaks flake8 too, which uses
.get().Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.7.14/x64/bin/flake8", line 8, in <module> sys.exit(main()) File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main app.run(argv) File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 363, in run self._run(argv) File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 350, in _run self.initialize(argv) File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 330, in initialize self.find_plugins(config_finder) File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 153, in find_plugins self.check_plugins = plugin_manager.Checkers(local_plugins.extension) File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 357, in __init__ self.namespace, local_plugins=local_plugins File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 238, in __init__ self._load_entrypoint_plugins() File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 254, in _load_entrypoint_plugins eps = importlib_metadata.entry_points().get(self.namespace, ()) AttributeError: 'EntryPoints' object has no attribute 'get'Reacted by Jacob Stevens-Haas, Lokman Rahmani and Neil MurphyThat's correct. Support for dict-based access was deprecated in
importlib_metadata3.6 (and Python 3.10) and intentionally removed in the 5.0 release (and planned for the Python 3.12 release).Is it possible the usage in stevedore didn't trigger the deprecation warnings?
Breaks flake8 too, which uses
.get().…though it looks like that was intentional looking at 7e5bae4, which removed a previously-deprecated interface and corresponding warnings silencer specifically for flake8.
The corresponding warnings silencer for flake8 was removed in #319. The only reason it shows up in that specific commit is because I made these removal commits against v4.4.0 to ensure that I wasn't removing anything that wasn't part of Python 3.10. The actual diff for this change can be seen in #405.
It's my understanding that later versions of flake8 provide a workaround for this issue but also that flake8 pins against and older version of importlib_metadata, so shouldn't get this behavior anyway. Any idea why an older version of flake8 is ending up with a brand new version of importlib_metadata?
Any idea why an older version of flake8 is ending up with a brand new version of importlib_metadata?
Other things in the dep chain need a newer importlib_metadata, so pip walks back the flake8 versions until it finds flake8 3.9.2 which is the last release that includes
importlib_metadatawithout a version pin. It doesn't work, of course, but pip can't know that.If I manually pin to flake8 >=4.0.0, the first release which includes the importlib_metadata pin, then pip instead walks back the versions of other things to a point where the version of importlib_metadata they need is compatible with the older version flake8 needs. (I'm not sure yet if this is acceptable or not for our usage, but at least it's something.)
Reacted by Clif BratcherOther things in the dep chain need a newer importlib_metadata, so pip walks back the flake8 versions until it finds flake8 3.9.2 which is the last release that includes
importlib_metadatawithout a version pin.Oh, interesting. That makes sense now that you explain it.
I'm not sure yet if this is acceptable or not for our usage
I'd expect that to be the case, as
importlib_metadatais mostly compatible between 3.6 and 5.0.I do believe that flake8 has removed the usage of the deprecated code paths in late releases, so they probably can remove the pin and help ease the pain.
If I manually pin to flake8 >=4.0.0, the first release which includes the importlib_metadata pin, then pip instead walks back the versions
That also seems like a reasonable workaround.
By the way, if it would help to yank the release and delay the break for a day or a week or a month to stop the bleeding, that’s certainly possible.
Workaround for poetry + flake8 + python 3.7+: This pin fixed the resolution for me
flake8 = [ { version = "*", python = "^3.7" }, { version = ">=5", python = "^3.8" }, ]
I truly don't know why the resolution has these issues. One possibility I haven't ruled out: https://github.com/PyCQA/flake8/blob/e94ee2b5f1801354b940cfe830b9160852915aec/setup.cfg#L45
Unfortunately importlib_metadata is used deep within many automated systems (build systems, containers, etc...), so the deprecation warnings are likely unseen by most and ignored by many others. It's super easy to see "this functionality is deprecated" and treat it as "I won't use it more in the future" instead of "I will stop using it now".
On that note, it might be helpful to have messaging regarding a date or version when breaking changes are coming. That way folks can plan for it, especially for a commonly used package.
I really appreciate the feedback.
On that note, it might be helpful to have messaging regarding a date or version when breaking changes are coming. That way folks can plan for it, especially for a commonly used package.
That's useful feedback. I've been operating under a less rigorous versioning strategy and I've only seen a few examples of projects that have adopted a rigorous strategy for breaking changes (Python recently with the two-version deprecation period, and pip with the breaking changes annually). I've found that different breaking changes have different blast radii, so it's difficult to devise a specific strategy that accommodates the variety of impacts. I try to ascertain the level of impact and use that to devise a timeline for deprecation/removal. Often, I will put a specific date in the code "not to be removed before YYYY-MM-DD" to provide downstream consumers a minimum time. I have found that attempting to say specifically when or in what version a breaking change will occur is folly and often gets missed.
I didn't put a particular date on this change in part because the deprecation itself was so impactful, I'd expected the major consumers to have already addressed the issue early and the 16 month delay gave sufficient time for systems to upgrade.
I do think it would have helped to put a minimum date on the deprecation warning. I'll do that for the future and possibly remaining deprecation warnings.
It's super easy to see "this functionality is deprecated" and treat it as "I won't use it more in the future" instead of "I will stop using it now".
That's an interesting perspective. I've never taken that perspective. Whenever I've encountered a
DeprecationWarning, that says to me that the functionality is slated for removal and could be removed at any time in any future version (unless stated otherwise). Interestingly, the Python documentation doesn't clarify what the meaning of aDeprecationWarningis.If some maintainers hold the opinion that
DeprecationWarningjust means limit additional usage, they will necessarily encounter the breakage every time it happens (even if it's years in the future). That is, unless no project ever changes any expectation ever.I could imagine some projects employing a
PendingDeprecationWarningto signal what you're suggesting, that further use be curtailed, but that existing uses are still okay for now... but it still implies that a deprecation is imminent and when that happens, one should take action and adapt.Per the Python deprecation policy, this behavior is expected to change in Python 3.12 as well, which is why I wanted to get the backport change out sooner, so users would have a chance to adapt where they have more control over the behavior (pinning dependencies, adding backports, etc).
Unfortunately importlib_metadata is used deep within many automated systems (build systems, containers, etc...), so the deprecation warnings are likely unseen by most and ignored by many others.
Good point. If there's a specific example where the directly-affected project (the project using the interface) did not have visibility to the deprecation warning, I'd like to investigate that, because that's an assumption I rely on to enact these changes. I know other projects downstream of the direct-consuming projects will be affected, and I rely on the direct consumers to take action when deprecations affect their projects and their users.
My current understanding is that the big projects affected are openstack (stevedore) and flake8, both projects with which I engaged directly at the time of the deprecation (or prior to it) and proposed multiple fixes to address the breakage early. If any project had reached out and requested that the removal be delayed, I certainly would have considered that and likely built it into the approach. Stability and predictability are important, but if a project chooses to ignore the warnings, they do so at their own risk (true of any project).
Reacted by Anja Kefala and 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко)- added a commit that references this issue
on Oct 4, 2022 20 remaining items
- added a commit that references this issue
on Oct 12, 2022 tuve el mismo problema en este caso con celery
esto se soluciona instalando en el proyecto la version de importlib-metadata==4.13.0
This was very painful but we are getting over it now.
- added 6 commits that reference this issue
on Oct 19, 2022 - added a commit that references this issue
on Oct 24, 2022 seconded. i don't have specific details like @inno but i am able to trace this back to upgrading to 5.0.0. the package that triggered the error was
celeryfor the time being i have downgraded to <5.0.0
- unpinned this issue
on Nov 5, 2022 - added a commit that references this issue
on Nov 8, 2022 Often, I will put a specific date in the code "not to be removed before YYYY-MM-DD" to provide downstream consumers a minimum time. I have found that attempting to say specifically when or in what version a breaking change will occur is folly and often gets missed.
I didn't put a particular date on this change in part because the deprecation itself was so impactful, I'd expected the major consumers to have already addressed the issue early and the 16 month delay gave sufficient time for systems to upgrade.
I think defining a specific version (maybe in addition to the date) would be more helpful. Unfortunately, a lot of python packages (including Celery, which caused a problem in my case) don't specify a maximum version/upper bound for their dependencies, but such warnings could help. (These would even help their users, i.e. us, if they don't include it regardless.)
The problem is that even if these dependent packages get updated before the change, if they don't use a max version in their pin and their users have to pin an earlier version of their package it will still break for them when you release a new version. Now I understand that this is mostly their fault, but my gut feeling is that stating a version number (as most projects I know do, BTW) will make it easier for them to handle it. Maybe even be a hint to define that pin...
entry_pointspreviously returned a dict-like object and other projects have taken advantage of this functionality (stevedore, for example, iterates via.items()). Asentry_pointsnow returns a list-like object, it breaks these previously working dict-like methods.Issue appears with this change: 7e5bae4