Skip to content

[FEA]: Ability to provide an extra path for cuda-pathfinder to search with high priority #1054

Closed
Enhancement
Closed
[FEA]: Ability to provide an extra path for cuda-pathfinder to search with high priority#1054
Enhancement
Assignees
rwgk
Labels
awaiting-responseFurther information is requestedcuda.pathfinderEverything related to the cuda.pathfinder modulefeatureNew feature or request
Milestone
cuda.pathfinder next

Description

@benhg
benhg
opened on Sep 30, 2025

Is this a duplicate?

Area

cuda.pathfinder

Is your feature request related to a problem? Please describe.

The cuda-pathfinder's search path prioritizes libraries from wheels. For example, libnvshmem_host.so comes from libnvidia-nvshmem-cuXX wheel. If I'm working from a development branch of nvshmem, I may need to manually copy libraries over the site-packages directory in order to get my custom libraries to be opened. LD_LIBRARY_PATH is much lower in the priority order (on purpose), so that won't work either.

Describe the solution you'd like

A few different ideas come to mind:

  • <LIBNAME_UPPER_CASE>_PATH_OVERRIDE so each library has an ability to be overridden independently
  • CUDA_PATHFINDER_TOP_PRIORITY_PATH which applies to all pathfinder searches
  • Adding a <library>_HOME environment variable, and prioritizing it above other search paths

Describe alternatives you've considered

I think I listed all 3 above. The current hacky WAR is to copy .so files around on the system.

Additional context

No response

Activity

  1. github-actions
    added
    triageNeeds the team's attention
    on Sep 30, 2025
  2. rwgk
    self-assigned this
    on Sep 30, 2025
  3. rwgk
    added
    enhancementAny code-related improvements
    cuda.pathfinderEverything related to the cuda.pathfinder module
    on Sep 30, 2025

  4. leofang commented on Sep 30, 2025

    @leofang
    leofang
    on Sep 30, 2025
    Member

    Why don't you just remove the wheel from the Python env, and let the system search kick in (through LD_LIBRARY_PATH)? This is not a typical request especially for library teams who own both C and Python libraries. If you don't keep the hygiene of your dev env, at one point things will just break and we can't be blamed.

  5. leofang
    added
    awaiting-responseFurther information is requested
    on Sep 30, 2025

  6. benhg commented on Sep 30, 2025

    @benhg
    benhg
    on Sep 30, 2025
    MemberAuthor

    Why don't you just remove the wheel from the Python env, and let the system search kick in (through LD_LIBRARY_PATH)? This is not a typical request especially for library teams who own both C and Python libraries. If you don't keep the hygiene of your dev env, at one point things will just break and we can't be blamed.

    We could certainly do that. It's just one extra step to make sure pip install never pulls back in those dependencies. I agree that it is very much a niche/developer-only feature and may not be worth doing if it only serves a small value.

  7. leofang commented on Sep 30, 2025

    @leofang
    leofang
    on Sep 30, 2025
    Member

    pip install --no-deps ... will help you avoid installing dependencies. Alternatively, you can organize non-nvSHMEM dependencies into a, say,requirements-dev-core.txt file, and then pip install -r requirements-dev-core.txt to just avoid nvSHMEM.

    I'd prefer us to not commit to any overwrite mechanisms for now (preferably forever). I don't want to repeat the nightmare of having RPATH vs RUNPATH and having to worry about the search precedence. The less factors participating in our search logic, the better.

  8. benhg commented on Sep 30, 2025

    @benhg
    benhg
    on Sep 30, 2025
    MemberAuthor

    Understandable. This is not a dealbreaker for us (or probably anyone else) if we can't have it.

  9. kkraus14 commented on Oct 1, 2025

    @kkraus14
    kkraus14
    on Oct 1, 2025
    Collaborator

    I agree we shouldn't reorder standard paths, but I agree with @benhg that there's value in having an "escape hatch" of sorts to override pathfinder behavior. This will inevitably come in handy when someone needs to pull in a debug build of a library or something else. I think if we have pathfinder specific environment variables that allow controlling on a per library and per component (library vs headers vs executables vs etc.) basis that it would be a valuable addition.

  10. rparolin
    added this to the cuda-pathfinder parking lot milestone on Oct 13, 2025
  11. rparolin
    removed
    triageNeeds the team's attention
    on Jan 19, 2026
  12. rwgk
    added
    featureNew feature or request
    and removed
    enhancementAny code-related improvements
    on Mar 4, 2026
  13. rparolin
    removed this from the cuda.pathfinder backlog milestone on Apr 2, 2026
  14. rparolin
    added the
    Enhancement
    issue type on Apr 26, 2026

  15. rparolin commented on Apr 27, 2026

    @rparolin
    rparolin
    on Apr 27, 2026
    Collaborator

    x-ref: #1979

  16. rparolin
    added this to the cuda.pathfinder next milestone on Aug 20, 2026

  17. rwgk commented on Aug 23, 2026

    @rwgk
    rwgk
    on Aug 23, 2026
    Contributor

    Hi @benhg, I wanted to call out that the just-merged PR #2680 includes features relevant to this issue.

    PR #2680 adds descriptor-driven product installation roots for cuDNN and NCCL. Existing Pathfinder APIs use these automatically for both dynamic-library and header discovery:

    • CUDNN_PATH=<root> supports the declared cuDNN layouts: lib/lib64 plus include on Linux; bin/x64 or bin plus include on Windows x64; and the verified standalone archive's bin/arm64 plus include layout on native Windows ARM64.
    • NCCL_HOME=<root> supports lib, lib64, and build/lib for the NCCL dynamic library on Linux, plus include and build/include for headers. This covers installed/extracted prefixes as well as an NCCL source/build tree.

    This resembles the third idea in this issue, but there is an important difference: these variables are conventional-layout fallbacks, not developer overrides.

    For dynamic libraries, the effective load precedence remains:

    1. An already-loaded library.
    2. A wheel or Conda candidate.
    3. Native OS loader search.
    4. The product root (CUDNN_PATH or NCCL_HOME).
    5. Generic CUDA roots and later fallbacks such as Program Files.

    For headers, wheel and Conda candidates likewise precede the product root; the product root then precedes generic CUDA and system locations. A product variable must point to an installation root with one of the declared layouts, and an unset, stale, or non-matching root falls through normally.

    So PR #2680 does not resolve the original nvSHMEM development use case here, mainly:

    • An nvSHMEM-specific product root is still missing.
    • Even an analogous product root at the current precedence would not supersede an installed nvSHMEM wheel.
    • It does not accept an arbitrary library file or directory as a highest-priority override.
    • It does not provide fail-loud semantics for a configured override that cannot be used.

    Do you think this issue should stay open for a genuine preemptive override, or would the current product-root model be sufficient if we added equivalent nvSHMEM metadata? It could let pathfinder discover a conventional nvSHMEM development or installation tree whenever no higher-priority wheel or Conda candidate is present.

  18. rparolin commented on Oct 29, 2026

    @rparolin
    rparolin
    on Oct 29, 2026
    Collaborator

    I'm going to close this issue due to inactivity. Feel free to re-open it again if you still need this functionality.

  19. rparolin
    closed this as completedon Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

awaiting-responseFurther information is requestedcuda.pathfinderEverything related to the cuda.pathfinder modulefeatureNew feature or request

Type

Enhancement

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions