`pip install --no-binary` fails on Python 3.7

[radoering]

When running pip install --no-binary :all: importlib-metadata or even pip install --no-binary importlib-metadata importlib-metadata on Python 3.7, building fails due to a recursion error. That's because the latest setuptools_scm version introduced a dependency on importlib-metadata for python_version < "3.8".

Maybe, build-system requires should be changed to require an older setuptools_scm version for python_version < "3.8"?

[jaraco]

Thanks for reporting this issue. I encountered something similar when attempting to add a dependency to importlib_metadata (#389 (comment)). In my opinion, the issue lies not with this package nor with setuptools_scm but with the fact that the Python packaging ecosystem can't resolve build dependencies when building from source (pypa/packaging-problems#342).

For what it's worth, I don't get a recursion error as reported. Instead, I get a "LookupError":

Details

$ docker run -it jaraco/multipy-tox py -3.7 -m pip install --force importlib_metadata --no-binary importlib_metadata
Collecting importlib_metadata
  Downloading importlib_metadata-4.11.4.tar.gz (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.2/44.2 kB 2.6 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [65 lines of output]
      Collecting setuptools>=56
        Downloading setuptools-62.6.0-py3-none-any.whl (1.2 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 17.9 MB/s eta 0:00:00
      Collecting setuptools_scm[toml]>=3.4.1
        Downloading setuptools_scm-7.0.2-py3-none-any.whl (40 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 kB 14.5 MB/s eta 0:00:00
      Collecting typing-extensions
        Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
      Collecting packaging>=20.0
        Using cached packaging-21.3-py3-none-any.whl (40 kB)
      Collecting tomli>=1.0.0
        Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
      Collecting importlib-metadata
        Using cached importlib_metadata-4.11.4.tar.gz (44 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
          status = run_func(*args)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/cli/req_command.py", line 205, in wrapper
          return func(self, options, args)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/commands/install.py", line 342, in run
          reqs, check_supported_wheels=not options.target_dir
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
          collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
          state = resolution.resolve(requirements, max_rounds=max_rounds)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
          failure_causes = self._attempt_to_pin_criterion(name)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
          criteria = self._get_updated_criteria(candidate)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
          self._add_to_criteria(criteria, requirement, parent=candidate)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
          if not criterion.candidates:
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
          return bool(self._sequence)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
          return any(self)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
          return (c for c in iterator if id(c) not in self._incompatible_ids)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
          candidate = func()
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/factory.py", line 220, in _make_candidate_from_link
          version=version,
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
          version=version,
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 161, in __init__
          self.dist = self._prepare()
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 230, in _prepare
          dist = self._prepare_distribution()
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 302, in _prepare_distribution
          return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/operations/prepare.py", line 428, in prepare_linked_requirement
          return self._prepare_linked_requirement(req, parallel_builds)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/operations/prepare.py", line 502, in _prepare_linked_requirement
          self.check_build_deps,
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/operations/prepare.py", line 57, in _get_prepared_distribution
          with build_tracker.track(req):
        File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
          return next(self.gen)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/operations/build/build_tracker.py", line 122, in track
          self.add(req)
        File "/tmp/pip-standalone-pip-oh2bbgg0/__env_pip__.zip/pip/_internal/operations/build/build_tracker.py", line 92, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/35/a8/f2bd0d488c2bf932b4dda0fb91cbb687c0b1132b33130d1cfad4e2b4b963/importlib_metadata-4.11.4.tar.gz#sha256=5d26852efe48c0a32b0509ffbc583fda1a2266545a78d104a6f4aff3db17d700 (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.7) is already being built: importlib_metadata from https://files.pythonhosted.org/packages/35/a8/f2bd0d488c2bf932b4dda0fb91cbb687c0b1132b33130d1cfad4e2b4b963/importlib_metadata-4.11.4.tar.gz#sha256=5d26852efe48c0a32b0509ffbc583fda1a2266545a78d104a6f4aff3db17d700
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[radoering]

You are right, I was a bit careless to call it a "recursion error". I agree that the real issue is deeper. My PR only presents a workaround. I'm not sure if that is acceptable.

[jaraco]

I'm struggling with what to do here. On one hand, the issue is rooted in the build dependencies (setuptools_scm), so in that sense, the build tool is closer to the root of the problem. It just so happens that because setuptools_scm narrowly adopted this library, the issue is narrowly scoped (only affects Python 3.7). But if setuptools_scm decided to adopt another dependency that built with setuptools_scm or needed a feature from later versions of importlib_metadata not available on Python 3.8 or 3.9, that would again break things and the scope here would need to be expanded. I'd really like to avoid this entanglement of concerns across projects.

On the other hand, this issue is currently very isolated and it's plausible it will just go away when support for Python 3.7 is dropped. In that case, the proposed workaround is a pragmatic solution.

The current stance of the PyPA (from pypa/packaging-problems#342) is that --no-binary is unsupported in these cases, so a part of me is inclined to leave this broken until that stance changes. It would help justify the case if I understood the impact of this issue to you and others. Can you explain more about the impact of the issue in your use-case? Why must you use --no-binary; could you workaround with --only-binary importlib_metadata? Are you an integrator or end user? Could you simply require Python 3.8+?

[radoering]

The issue popped up in a test case in poetry-core, which is a PEP 517 build backend. Unfortunately, I'm not familiar enough with the background to say if, let alone why, this is relevant to us. I pinged a more experienced maintainer who will hopefully be able to provide some details.

[jaraco]

I don't believe this issue is actionable here, so I'm closing it. Happy to revisit if there's something that importlib_metadata can do.

[Reonaydo]

Nothing changed. On python 3.7 it is impossible to install importlib-metadata with --no-binary flag

$ /tmp/venv/bin/pip install --no-binary :all: importlib-metadata
Collecting importlib-metadata
  Using cached importlib_metadata-6.7.0.tar.gz (53 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [77 lines of output]
      Collecting setuptools>=56
        Using cached setuptools-68.0.0-py3-none-any.whl
      Collecting setuptools_scm[toml]>=3.4.1
        Using cached setuptools_scm-7.1.0.tar.gz (71 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting packaging>=20.0 (from setuptools_scm[toml]>=3.4.1)
        Using cached packaging-23.1-py3-none-any.whl
      Collecting typing-extensions (from setuptools_scm[toml]>=3.4.1)
        Using cached typing_extensions-4.7.1-py3-none-any.whl
      Collecting tomli>=1.0.0 (from setuptools_scm[toml]>=3.4.1)
        Using cached tomli-2.0.1.tar.gz (15 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting importlib-metadata (from setuptools_scm[toml]>=3.4.1)
        Using cached importlib_metadata-6.7.0.tar.gz (53 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
          status = run_func(*args)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
          return func(self, options, args)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 378, in run
          reqs, check_supported_wheels=not options.target_dir
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 93, in resolve
          collected.requirements, max_rounds=limit_how_complex_resolution_can_be
        File "/tmp/venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
          state = resolution.resolve(requirements, max_rounds=max_rounds)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
          failure_causes = self._attempt_to_pin_criterion(name)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
          criteria = self._get_updated_criteria(candidate)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 230, in _get_updated_criteria
          self._add_to_criteria(criteria, requirement, parent=candidate)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
          if not criterion.candidates:
        File "/tmp/venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
          return bool(self._sequence)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
          return any(self)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
          return (c for c in iterator if id(c) not in self._incompatible_ids)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
          candidate = func()
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 211, in _make_candidate_from_link
          version=version,
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 299, in __init__
          version=version,
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
          self.dist = self._prepare()
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
          dist = self._prepare_distribution()
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
          return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 538, in prepare_linked_requirement
          return self._prepare_linked_requirement(req, parallel_builds)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 658, in _prepare_linked_requirement
          self.check_build_deps,
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 68, in _get_prepared_distribution
          with build_tracker.track(req):
        File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
          return next(self.gen)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/operations/build/build_tracker.py", line 122, in track
          self.add(req)
        File "/tmp/venv/lib/python3.7/site-packages/pip/_internal/operations/build/build_tracker.py", line 92, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/a3/82/f6e29c8d5c098b6be61460371c2c5591f4a335923639edec43b3830650a4/importlib_metadata-6.7.0.tar.gz (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.7) is already being built: importlib-metadata from https://files.pythonhosted.org/packages/a3/82/f6e29c8d5c098b6be61460371c2c5591f4a335923639edec43b3830650a4/importlib_metadata-6.7.0.tar.gz
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.