[AutoSparkUT] Migrate DataFramePivotSuite tests to RAPIDS#13761
Merged
wjxiz1992 merged 2 commits intoNVIDIA:mainfrom Nov 14, 2025
Merged
[AutoSparkUT] Migrate DataFramePivotSuite tests to RAPIDS#13761wjxiz1992 merged 2 commits intoNVIDIA:mainfrom
wjxiz1992 merged 2 commits intoNVIDIA:mainfrom
Conversation
- Created RapidsDataFramePivotSuite extending DataFramePivotSuite with RapidsSQLTestsBaseTrait - Registered test suite in RapidsTestSettings.scala - All 31 tests passing successfully with no exclusions needed: * pivot courses * pivot year * pivot courses with multiple aggregations * pivot year with string values (cast) * pivot year with int values * pivot courses with no values * pivot year with no values * pivot max values enforced * pivot with UnresolvedFunction * optimized pivot planned * optimized pivot courses with literals * optimized pivot year with literals * optimized pivot year with string values (cast) * optimized pivot DecimalType * PivotFirst supported datatypes * optimized pivot with multiple aggregations * pivot with datatype not supported by PivotFirst * pivot with datatype not supported by PivotFirst 2 * pivot preserves aliases if given * pivot with column definition in groupby * pivot with null should not throw NPE * pivot with null and aggregate type not supported by PivotFirst returns correct result * pivot with timestamp and count should not print internal representation * SPARK-24722: pivoting nested columns * SPARK-24722: references to multiple columns in the pivot column * SPARK-24722: pivoting by a constant * SPARK-24722: aggregate as the pivot column * pivoting column list with values * SPARK-26403: pivoting by array column * SPARK-35480: percentile_approx should work with pivot * SPARK-38133: Grouping by TIMESTAMP_NTZ should not corrupt results - Perfect compatibility with GPU execution - no issues found This is an excellent test suite with 100% pass rate! Signed-off-by: Allen Xu <wjxiz1992@gmail.com>
wjxiz1992
force-pushed
the
dataframe-pivot-suite-migration
branch
from
80000e5 to
a36fe8c
Compare
Contributor
Greptile OverviewGreptile SummaryMigrates Apache Spark's
Confidence Score: 5/5
Important Files ChangedFile Analysis
Sequence DiagramsequenceDiagram
participant Dev as Developer
participant Suite as RapidsDataFramePivotSuite
participant Base as DataFramePivotSuite
participant Trait as RapidsSQLTestsBaseTrait
participant GPU as RAPIDS GPU Engine
Dev->>Suite: Run test suite
Suite->>Base: Inherit pivot tests
Suite->>Trait: Mix in GPU configuration
Trait->>Trait: Configure Spark with RAPIDS plugin
Trait->>Trait: Enable GPU acceleration settings
Base->>Suite: Execute inherited tests
Suite->>GPU: Run tests on GPU
GPU->>Suite: Return results
Suite->>Dev: All 31 tests pass
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Collaborator
Author
|
build |
thirtiseven
approved these changes
Nov 14, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR migrates the
DataFramePivotSuitetest suite from Apache Spark to RAPIDS Accelerator for Apache Spark.Test Results
✅ Perfect Migration - 100% Pass Rate!
All tests pass successfully on GPU without any modifications or exclusions needed!
Test Coverage
The migrated tests cover:
Implementation Details
RapidsDataFramePivotSuiteextendingDataFramePivotSuitewithRapidsSQLTestsBaseTraitRapidsTestSettings.scalaRelated Issues
Part of #11297 (Spark unit test migration effort)
Testing
Result:
All tests passed.Signed-off-by: Allen Xu wjxiz1992@gmail.com