Work around nvcc 13.0 parse failure in group_quantize_fp8.cuh

[xiuhu17]

Scenario

Building with CUDA 13.0 (nvcc V13.0.88, sm100) fails in every TU that includes group_quantize_fp8.cuh (e.g. activation/glu.cu):

error: template argument 2 is invalid
error: 'IVecT' has not been declared

Root cause is an nvcc 13.0 cudafe++ bug: inside the macro-expanded TRANSFORMER_ENGINE_*_SWITCH block, the discarded if constexpr branch constant-folds the constexpr size_t flat_nvec template argument and emits an ill-formed two-token functional cast unsigned long((8)) into the intermediate C++, which the host compiler then rejects. Reproduces with both g++-12 and g++-13 hosts; the identical code hand-expanded (no macro) compiles fine. Confirmed with a 29-line minimal repro.

Fix

Replace the using IVecT/OVecT = Vec<IType/OType, flat_nvec> aliases (only used for ::BYTES in the alignment checks) with the equivalent constexpr size_t byte counts (Vec<T, N>::BYTES == N * sizeof(T), utils.cuh). Semantics-identical, host-side only, no kernel change. Verified: sm100 wheel builds clean on CUDA 13.0 and the NVFP4/FP8 paths run end-to-end.

[greptile-apps]

Greptile Summary

This PR works around an nvcc 13.0 (CUDA 13.0) cudafe++ parse bug that causes compilation failures in any translation unit including group_quantize_fp8.cuh. Inside the TRANSFORMER_ENGINE_*_SWITCH macro-expanded blocks, nvcc incorrectly constant-folds the constexpr size_t flat_nvec template argument and emits an ill-formed two-token cast into the intermediate C++, which then fails host compilation.

• The fix removes using IVecT = Vec<IType, flat_nvec> and using OVecT = Vec<OType, flat_nvec> from two if constexpr (SCALING_TYPE == ScalingType::ROWWISE) branches where they were only used for their ::BYTES value, replacing them with constexpr size_t flat_in_vec_bytes = flat_nvec * sizeof(IType) and constexpr size_t flat_out_vec_bytes = flat_nvec * sizeof(OType) — values provably identical to Vec<T, N>::BYTES per the definition in utils.cuh.
• All other IVecT/OVecT usages in this file live inside GPU kernel bodies (not host-dispatch macro blocks) and are completely unaffected.

Confidence Score: 5/5

This is a safe, minimal workaround that changes only host-side alignment checks; kernel logic and data paths are untouched.

The only changed code is two pairs of using alias declarations replaced with mathematically equivalent constexpr size_t expressions, verified against the Vec definition in utils.cuh. No kernel code is altered, no runtime behavior changes, and the fix is confined to the exact two sites that triggered the nvcc 13.0 bug.

No files require special attention.

Important Files Changed

Filename	Overview
transformer_engine/common/cast/fp8/group_quantize_fp8.cuh	Replaces two using IVecT/OVecT type aliases (only used for ::BYTES) inside macro-expanded host dispatch blocks with semantically-identical constexpr size_t byte counts to work around an nvcc 13.0 cudafe++ parse bug; all other uses of these aliases in GPU kernels are untouched.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["group_quantize() host dispatch"] --> B["TRANSFORMER_ENGINE_*_SWITCH macro"]
    B --> C{"SHAPE_REP == SAME_BOTH_DIMS?"}
    C -->|yes| D{"SCALING_TYPE == ROWWISE?"}
    D -->|yes| E["Before: using IVecT/OVecT\nAfter: constexpr size_t flat_in/out_vec_bytes"]
    E --> F["flat_aligned check using % bytes"]
    F --> G["Launch flat kernel or skip"]
    C -->|no| H{"SHAPE_REP == VARYING_FIRST_DIM?"}
    H -->|yes| I{"SCALING_TYPE == ROWWISE?"}
    I -->|yes| J["Before: using IVecT/OVecT\nAfter: constexpr size_t flat_in/out_vec_bytes"]
    J --> K["flat_aligned check using % bytes"]
    K --> L["NVTE_CHECK + launch kernel"]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["group_quantize() host dispatch"] --> B["TRANSFORMER_ENGINE_*_SWITCH macro"]
    B --> C{"SHAPE_REP == SAME_BOTH_DIMS?"}
    C -->|yes| D{"SCALING_TYPE == ROWWISE?"}
    D -->|yes| E["Before: using IVecT/OVecT\nAfter: constexpr size_t flat_in/out_vec_bytes"]
    E --> F["flat_aligned check using % bytes"]
    F --> G["Launch flat kernel or skip"]
    C -->|no| H{"SHAPE_REP == VARYING_FIRST_DIM?"}
    H -->|yes| I{"SCALING_TYPE == ROWWISE?"}
    I -->|yes| J["Before: using IVecT/OVecT\nAfter: constexpr size_t flat_in/out_vec_bytes"]
    J --> K["flat_aligned check using % bytes"]
    K --> L["NVTE_CHECK + launch kernel"]

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into zhw/fix_compile" | Re-trigger Greptile}

[vthumbe1503]

LGTM

[vthumbe1503]

/te-ci

[xiuhu17]

@vthumbe1503 @ptrendx Thanks for reviewing! Could anyone help to merge the pr? Thanks!

[ptrendx]

Merged. Thank you @xiuhu17 !

[xiuhu17]

@ptrendx Thanks for the help!