`compile_pip_requirements` `.update` does not work with local sub modules

[faximan]

🐞 bug report

Affected Rule

The issue is caused by the rule: compile_pip_requirements

Is this a regression?

No

Description

When using a nested Bazel submodule, compile_pip_requirements fails to find and write requirements.txt as the path component containing the module is stripped out.

https://github.com/bazel-contrib/rules_python/pull/1124/files#diff-92abbbd273b0a63585ad40be538589b70bf6468143f0cf37152fc89b28afc526 added most of this logic (follow the resolved_requirements_txt var).

🔬 Minimal Reproduction

https://github.com/faximan/requirements-repro (see README.md)

🔥 Exception or Error

bazel --bazelrc=/dev/null run @subrepo//:requirements.update
Updating requirements.txt
Traceback (most recent call last):
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/_main/../rules_python+/python/private/pypi/dependency_resolver/dependency_resolver.py", line 238, in 
    main()
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/rules_python++internal_deps+pypi__click/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/faximan/.cache/bazel/_bazel_faximan/9967d5615e05634f42c83c0e87c20173/execroot/_main/bazel-out/k8-fastbuild/bin/external/subrepo+/requirements.update.runfiles/_main/../rules_python+/python/private/pypi/dependency_resolver/dependency_resolver.py", line 176, in main
    os.makedirs(os.path.dirname(requirements_file_relative), exist_ok=True)
  File "", line 225, in makedirs
FileNotFoundError: [Errno 2] No such file or directory: ''

🌍 Your Environment

Operating System:

  
Linux
  

Output of bazel version:

  
8.1.1
  

Rules_python version:

  
1.3.0
  

Anything else relevant?
faximan#1 fixes the issue for me, but probably in the wrong way. I don't think writing directly to the runfiles tree is necessarily the right thing to do? It also currently leaves behind some .__atomic-write files.

Since I am not familiar with this codebase, behavior on Windows etc. I'm looking for someone else to take this over and propose a PR.

[aignas]

Having gone through designing the new lock rule that will be available to users in the next release (#2657) I would say that supporting the use case reliably and covering all edge cases is somewhat hard.

This does not work in:

• the case where you want to write the file for the first time.
• the case where the submodule is somewhere deeper - bazel just does not give you the actual path that you need to use here.

Hence I am inclined to close this as won't do unless there is a very compelling reason why not to. Just so that I know exactly the use case that you have, could you please explain in more detail what kind of workflows this would enable?

[faximan]

Sorry, I did not see your comment until now. Thanks for having a look!

Our use case is granted somewhat odd: we have a monorepo setup with a root bazel module. Then we have a subfolder which is authored outside of the monorepo in an environment that does not have Bazel, but regularly imported into the monorepo. This subfolder is its own bazel module.

Users of the monorepo often need to build targets inside the subrepo, but they almost always do so from the root (bazel build @subrepo//my:target), so this is what Bazel will have cached, what is tested in CI etc. The requirements targets happen to live in the subrepo, but now requires cd subrepo; bazel run //:requirements.update which is then suboptimal.

Re your edge cases: our team does not care about writing the file for the first time or deeper nesting. It is probably not a compelling of a use case that you're interested in, but I think it is still a valid FR (especially the error msg is unclear).

[rickeylev]

Quick drive by comment

Short version: this can't be made to generally work because a submodule may not be writable.

Long version: to make it work, a client-local path has to be passed into the pip_compile rule somehow. Not sure how that could be done in a general way. Maybe tricks during the bzlmod phase with a custom extension? Not sure.

Re: run files modification: yeah, that isn't kosher. It's only working because they happen to be symlinks back to the original source, but that isn't guaranteed by bazel