feat: updates to the py_extension rule for building C extensionsPy extension by rsartor-cmd · Pull Request #3875 · bazel-contrib/rules_python

rsartor-cmd · 2026-06-30T04:14:48Z

Per feedback from #3851, this PR re-works the platform detection logic to rely on platform constraints instead of the Python runtime/toolchain.

Description

This PR contains updates to the experimental py_extension implementation. It re-works the platform tag detection to rely on modern platform constraints, and refactors the compilation and linking mechanism
to delegate to Bazel's native cc_shared_library and cc_library rules.

This PR targets the main repository's py-extension branch (not main).

Key Changes

1. Platform Tag & ABI Tag Derivation

Introduced abi_tag to PyCcToolchainInfo: Added the abi_tag field to the PyCcToolchainInfo provider, populated by the py_cc_toolchain rule. It defaults to deriving the tag from
python_version (e.g. cpython-311) for backward compatibility.
Constraint-Based Detection: Replaced legacy parsing of cc_toolchain CPU names with direct lookup in rules_python's central PLATFORMS registry using platform constraints.
Limited API Configuration: Updated how limited API (.abi3.so) suffixes are appended based on the toolchain configuration.

2. Compilation & Linking Delegation (Refactor to `cc_shared_library`)

Macro Fusion: Replaced the custom linking logic in the py_extension rule with a macro of the same name. It now accepts C/C++ source/header files directly (srcs, hdrs, copts, defines),
implicitly wrapping them in a private cc_library under the hood.
Consolidated Attributes: Consolidated static linkage under standard deps (removing the redundant static_deps attribute) and aligned linker arguments with cc_shared_library's user_link_flags.
Wrapper Rule (_py_extension_wrapper): Added a lightweight, private rule that wraps the cc_shared_library output to:
Query the Python toolchain for platform/ABI tags to resolve PEP 3149 compliant filenames (e.g. .cpython-311-x86_64-linux-gnu.so or .abi3.so).
Create a cheap symlink from the CSL output to the PEP 3149 name.
Propagate PyInfo (for python rules) and CcSharedLibraryInfo (for dynamic C++ dependencies).
ODR Validation: Leveraging native cc_shared_library ensures strict analysis-time validation against One Definition Rule (ODR) violations (e.g., duplicate static linkage in dynamic chains).

3. Runfiles and Data Support

Runfiles Propagation: Re-implemented runfiles collection using the repository's standard builders.RunfilesBuilder to support runtime assets (data attribute) and dynamic library dependency
propagation.

Verification & Testing

Added parity test cases in dependency_graph_tests.bzl to verify dynamic dependency chains and static sharing behavior.
Added runfiles validation in py_extension_tests.bzl to verify data asset propagation.
Updated limited API tests to match CSL output structure.
All 14 tests in //tests/cc/py_extension/... are compiling and passing.
──────
TAG=agy
CONV=23f8a5e8-2d99-401a-9903-1256b7d42b0e

…me toolchain.

…tions and may produce false-positives.

gemini-code-assist

Code Review

This pull request refactors the py_extension rule to use constraint-based platform tag derivation instead of C++ toolchain-based derivation, introduces abi_tag to PyCcToolchainInfo, and removes the _check_limited_api_compatibility validation logic. The review feedback highlights several critical issues: first, the new constraint-based platform detection is broken for musl Linux platforms because they share the same constraints as gnu platforms; second, changing the limited API check to bool(ctx.attr.py_limited_api) breaks backward compatibility for targets explicitly setting "none"; third, the hardcoded _constraints list should be dynamically derived from PLATFORMS to prevent future maintenance issues; and finally, the documentation for py_limited_api needs to be updated since the dependency validation logic was removed.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

rsartor-cmd · 2026-06-30T16:16:53Z

The mypy CI failure appears to be caused by the check action using Python 3.9, but the newest mypy only supporting 3.10+

Run docker build /home/runner/work/_actions/jpetrucciani/mypy-check/master --build-arg "pyver=3.9" -t mypy

#11 3.773 ERROR: Ignored the following versions that require a different python version: 1.20.0 Requires-Python >=3.10; 1.20.1 Requires-Python >=3.10; 1.20.2 Requires-Python >=3.10; 2.0.0 Requires-Python >=3.10; 2.1.0 Requires-Python >=3.10
#11 3.774 ERROR: Could not find a version that satisfies the requirement mypy==2.1.0 (from versions: 0.1, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.221, 0.222, 0.223, 0.224, 0.225, 0.226, 0.227, 0.228, 0.229, 0.230, 0.231, 0.232, 0.233, 0.234, 0.235, 0.236, 0.237, 0.238, 0.239, 0.240, 0.241, 0.250, 0.251, 0.252, 0.253, 0.254, 0.255, 0.256, 0.470, 0.471, 0.501, 0.510, 0.511, 0.520, 0.521, 0.530, 0.540, 0.550, 0.560, 0.570, 0.580, 0.590, 0.600, 0.610, 0.620, 0.630, 0.641, 0.650, 0.660, 0.670, 0.700, 0.701, 0.710, 0.711, 0.720, 0.730, 0.740, 0.750, 0.760, 0.761, 0.770, 0.780, 0.781, 0.782, 0.790, 0.800, 0.812, 0.900, 0.901, 0.902, 0.910, 0.920, 0.921, 0.930, 0.931, 0.940, 0.941, 0.942, 0.950, 0.960, 0.961, 0.971, 0.981, 0.982, 0.990, 0.991, 1.0.0, 1.0.1, 1.1.1, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.7.0, 1.7.1, 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.11.1, 1.11.2, 1.12.0, 1.12.1, 1.13.0, 1.14.0, 1.14.1, 1.15.0, 1.16.0, 1.16.1, 1.17.0, 1.17.1, 1.18.1, 1.18.2, 1.19.0, 1.19.1)
#11 3.775 ERROR: No matching distribution found for mypy==2.1.0
#11 ERROR: process "/bin/sh -c pip install -U pip &&     pip install -r requirements.txt" did not complete successfully: exit code: 1

For what it's worth, I ran mypy --show-column-numbers --hide-error-context python/runfiles and mypy --show-column-numbers --hide-error-context tests/runfiles locally and it found no issues with 3 files.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

… of hard-coding.

…nfirm py_extension behaves similarly to cc_shared_library.

…ry and cc_shared_library targets.

mkruskal-google · 2026-07-07T00:12:50Z

+    )
+
+def _csl_dynamic_deps_test_impl(env, target):
+    env.expect.that_target(target).has_provider(CcSharedLibraryInfo)


This test seems a little light. Don't we need to check more to ensure it actually linked everything correctly? Same for the others

Yes, I can add tests for that. These tests were in the earlier stages, when I was working out what providers were necessary to return.

@mkruskal-google Done. How does that look? eb4b8a8

rickeylev · 2026-06-30T18:45:08Z

 py_extension(
    name = "ext_limited",
+    py_limited_api = "3.8",
    static_deps = [":ext_limited_impl"],


please rename static_deps to deps (can be done separate pr)

rickeylev · 2026-07-09T03:21:42Z

            imports = depset([import_path]),
        ),
-        propagated_cc_info,
+        csl_target[CcSharedLibraryInfo],


Why do we propagate CcSharedLibraryInfo? It's so that the py extension can be given as a dynamic dep to another cc shared library? I'm not sure that makes sense -- lets discuss in chat.

rickeylev · 2026-07-09T03:25:42Z

-    ],
+    # An extension that also depends on a data file
+    name = "ext_with_data",
+    data = ["test_symbols.h"],


Lets use a non-code type of file for a data dependency, i.e. somedata.txt

rickeylev · 2026-07-09T03:35:32Z

+#####
+
+py_extension(
+    # An extension with its PyInit_* function in a dynamic dependency


Don't support this case, or rather, don't add a test for it -- it doesn't make any sense. The PyInit symbol is supposed to be provided by the foo.so module being loaded. If its provided by a shared library, that would imply that others are supposed to load that shared library and also get that symbol -- but that symbol is supposed to be unique to the loaded foo.so module

rickeylev · 2026-07-09T03:40:00Z

+    "_constraints": lambda: attrb.LabelList(
+        default = sorted({
+            c: None
+            for info in PLATFORMS.values()


The PLATFORMs global is legacy and shouldn't be used. The main problem is it requires every platform to be defined in rules_python, but that doesn't scale.

rickeylev · 2026-07-09T03:54:48Z

+
+    # 4. Create the underlying cc_shared_library
+    csl_name = "_" + name + "_csl"
+    csl_kwargs = {}


There's a small collection of attributes (visiblity, compatible_with, some others) that need to be copied into depended-upon generated targets. I think there's a helper function in utils for this; a todo for now is fine.

rickeylev · 2026-07-09T03:56:00Z

            files = depset([py_dso]),
            runfiles = runfiles,
        ),
        PyInfo(


Also set the "has shared libraries" field (I forget its exact name)

rickeylev · 2026-07-09T03:59:10Z

-        user_link_flags.append("-Wl,--allow-shlib-undefined")
+    import_path = repo_name
+    if ctx.label.package:
+        import_path = repo_name + "/" + ctx.label.package


Setting import path here is incorrect. It's going to add the directory the .so file is in to the python sys.path search, which makes it directly importable. However, this is wrong for two reasons:

A foo.so file can be part of package. In whih case, import pkg.foo is how to import it, and import foo would be incorrect

If it is part of package, all py files in the directory become directly importable as top-level modules. e.g foo/bar.py becomes importable as import bar. This is incorrect, as it breaks the identify modules have, causes duplicate code loading, and can shadow other top-level imports.

rickeylev · 2026-07-09T04:03:37Z


+    abi_tag = ctx.attr.abi_tag
+    if not abi_tag:
+        # Derive default: cpython-XX


This calculation is incomplete: it also needs to take into account freethreadedness.

You can copy how py_runtime computes it. I think it has an abiflags attribute? I can't remember exactly.

rickeylev · 2026-07-09T04:10:41Z

+        kwargs["data"] = data
+
+    # 6. Wrap with py_extension_wrapper for PEP 3149 naming & PyInfo
+    py_extension_wrapper(


Since we now have an internal-only wrapper rule we can pass extra arbitrary internal-use-only args, I think we can simplify how the platform info is passed in: have os and arch attributes, then use select. This also makes it easy to map bazel's conditions to the necessary python value

os = select({ "@platforms//os:windows": "win", "@platforms//os:osx": "darwin", }) arch = select({ "@platforms//cpu:x86_64": ..., ... })

rsartor-cmd added 15 commits June 29, 2026 18:58

Don't leave a trailing slash for the root package.

8214422

Remove duplicate assignment.

1e89b99

Remove typical C-style integer suffixes before parsing.

d921d20

Fix check for filename.

f318173

Remove unnecessary guard.

80a9aae

Re-format docstring.

9bfc331

Get the platform from the constraints.

e85e06b

Adjust formatting syntax.

590b667

Change the default value.

15c95f8

Add more test cases for py_limited_api.

daa2455

Use RunfilesBuilder instead of manual.

030c879

Add abi_tag to PyCcToolchainInfo.

73eb7c0

Use the PyCcToolchainInfo to derive the filename instead of the runti…

5cb7c90

…me toolchain.

Remove the fallback. Determine platform tag solely from constraints.

797bf98

Remove the version compatibility check, as it has performance implica…

09a2e0a

…tions and may produce false-positives.

rsartor-cmd requested review from aignas and rickeylev as code owners June 30, 2026 04:14

gemini-code-assist Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread python/private/cc/py_extension_rule.bzl

Comment thread python/private/cc/py_extension_rule.bzl

Comment thread python/private/cc/py_extension_rule.bzl

Comment thread python/private/cc/py_extension_rule.bzl Outdated

Fix header filename.

5bbe09f

rsartor-cmd and others added 10 commits June 30, 2026 11:29

Update python/private/cc/py_extension_rule.bzl

475ed3d

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

ruff

903c96a

Derive the values for the _constraints attr programmatically, instead…

9570a98

… of hard-coding.

Take glibc-vs-musl into account, so we get the right platform and name.

7179c8a

Clean up as per the linter.

b353be6

Remove print()s per buildifier.

ee6395d

Remove unused variable (buildifier).

8d80783

Sort keys [buildifier]

451ea48

Remove print()s per buildifier.

a40ae2b

Remove unused load().

90ad765

rsartor-cmd added 6 commits June 30, 2026 17:09

Call out unused parameters [buildifier]

708dac4

Add some basic tests of different dependency situations, so we can co…

830723f

…nfirm py_extension behaves similarly to cc_shared_library.

Replace the custom compiling and linking logic with internal cc_libra…

37f5eaa

…ry and cc_shared_library targets.

Add missing files.

4596ae2

Add a data argument to the macro.

933f69f

Add missing file

6f05ae0

mkruskal-google reviewed Jul 7, 2026

View reviewed changes

rsartor-cmd added 5 commits July 7, 2026 16:36

Add a couple more examples for PyInit_* defined in dependency targets.

8f2fa54

Add missing files

908ccd5

Expand the analysis tests with more details assertions.

eb4b8a8

Re-format files

12bc59e

Address buildifier and buildifier-lint warnings.

b158975

rickeylev reviewed Jul 9, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

rsartor-cmd commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

1. Platform Tag & ABI Tag Derivation

2. Compilation & Linking Delegation (Refactor to cc_shared_library)

3. Runfiles and Data Support

Verification & Testing

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rsartor-cmd commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rsartor-cmd commented Jun 30, 2026 •

edited

Loading

2. Compilation & Linking Delegation (Refactor to `cc_shared_library`)

rsartor-cmd commented Jun 30, 2026 •

edited

Loading