Skip to content

fix: keep join filter columns when pushing a projection below SortMergeJoinExec #24593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jayzhan211 merged 3 commits into from
Aug 26, 2026
Merged

fix: keep join filter columns when pushing a projection below SortMergeJoinExec #24593

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4aeb700
fix: keep join filter columns when pushing a projection below SortMer…
jayzhan211 Aug 23, 2026
b8fdebc
Merge branch 'main' of github.com:apache/datafusion into fix/smj-proj…
jayzhan211 Aug 24, 2026
c1bf726
test: enhance projection pushdown filter tests with additional cases
jayzhan211 Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 37 additions & 67 deletions datafusion/physical-plan/src/joins/sort_merge_join/exec.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,8 @@ use crate::joins::utils::{
};
use crate::metrics::{ExecutionPlanMetricsSet, MetricsSet, SpillMetrics};
use crate::projection::{
EmbeddedProjection, ProjectionExec, join_allows_pushdown, join_table_borders,
new_join_children, physical_to_column_exprs, try_embed_projection,
update_join_filter, update_join_on,
EmbeddedProjection, JoinData, ProjectionExec, try_embed_projection,
try_pushdown_through_join_with_column_indices,
};
use crate::spill::spill_manager::SpillManager;
use crate::statistics::{ChildStats, StatisticsArgs};
Expand Down Expand Up @@ -707,82 +706,53 @@ impl ExecutionPlan for SortMergeJoinExec {
}))
}

/// Tries to swap the projection with its input [`SortMergeJoinExec`]. If it can be done,
/// it returns the new swapped version having the [`SortMergeJoinExec`] as the top plan.
/// Otherwise, it returns None.
/// Tries to push `projection` down through this join. If possible, returns a
/// new [`SortMergeJoinExec`] whose children are the projected inputs. Otherwise
/// the join applies the projection itself (see [`EmbeddedProjection`]).
fn try_swapping_with_projection(
&self,
projection: &ProjectionExec,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
if self.projection.is_some() {
return Ok(None);
}
// Convert projected PhysicalExpr's to columns. If not possible, we cannot proceed.
let Some(projection_as_columns) = physical_to_column_exprs(projection.expr())
else {
return Ok(None);
};

let (far_right_left_col_ind, far_left_right_col_ind) = join_table_borders(
self.left().schema().fields().len(),
&projection_as_columns,
let schema = self.schema();
let (_, column_indices) = build_join_schema(
&self.left().schema(),
&self.right().schema(),
&self.join_type,
);

// Pushing into the children needs each side's columns to stay together, which
// an arbitrary projection does not. The join can apply that one itself.
if !join_allows_pushdown(
&projection_as_columns,
&self.schema(),
far_right_left_col_ind,
far_left_right_col_ind,
) {
return try_embed_projection(projection, self);
}

let left_field_size = self.left().schema().fields().len();
let left_projection = &projection_as_columns[0..=far_right_left_col_ind as usize];
let right_projection = &projection_as_columns[far_left_right_col_ind as usize..];

let Some(new_on) = update_join_on(
left_projection,
right_projection,
// Remaps the join keys and the filter's column indices to the
// projected children, and declines the pushdown if the projection
// drops a column the filter needs.

@jayzhan211 jayzhan211 Aug 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overall idea of this PR

if let Some(JoinData {
projected_left_child,
projected_right_child,
join_filter,
join_on,
}) = try_pushdown_through_join_with_column_indices(

@jayzhan211 jayzhan211 Aug 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try_pushdown_through_join_with_column_indices includes what we have for the old code

@kosiew kosiew Aug 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement here. One optional thought: could we add an execution-level unit test for the successful remapping path, where the filter columns are retained but reordered? The SQL regression covers the fallback with an embedded projection, while the current unit test verifies the rewritten indices without executing the returned SortMergeJoinExec. Executing it would give us a bit more confidence that the remapped JoinFilter schema and columns are consumed correctly. The broader SortMergeJoin spill SQL tests already exercise filtered projected joins, so I see this as a small coverage improvement rather than something that should block this PR.

@jayzhan211 jayzhan211 Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review, test added!

projection,
self.left(),
self.right(),
self.on(),
left_field_size,
) else {
return try_embed_projection(projection, self);
};

let new_filter = if let Some(filter) = self.filter() {
let Some(filter) = update_join_filter(
left_projection,
right_projection,
filter,
left_field_size,
) else {
return try_embed_projection(projection, self);
};
Some(filter)
&schema,
self.filter().as_ref(),
&column_indices,
)? {
Ok(Some(Arc::new(SortMergeJoinExec::try_new(
Arc::new(projected_left_child),
Arc::new(projected_right_child),
join_on,
join_filter,
self.join_type,
self.sort_options.clone(),
self.null_equality,
)?)))
} else {
None
};

let (new_left, new_right) = new_join_children(
&projection_as_columns,
far_right_left_col_ind,
far_left_right_col_ind,
self.children()[0],
self.children()[1],
)?;

Ok(Some(Arc::new(SortMergeJoinExec::try_new(
Arc::new(new_left),
Arc::new(new_right),
new_on,
new_filter,
self.join_type,
self.sort_options.clone(),
self.null_equality,
)?)))
try_embed_projection(projection, self)
}
}

#[cfg(feature = "proto")]
Expand Down
62 changes: 55 additions & 7 deletions datafusion/physical-plan/src/joins/sort_merge_join/tests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,8 +365,16 @@ async fn join_collect_batch_size_equals_two(
fn join_and_projection_for_pushdown(
filter: Option<JoinFilter>,
) -> Result<(Arc<SortMergeJoinExec>, ProjectionExec)> {
let left = build_table(("a1", &vec![1]), ("b1", &vec![2]), ("c1", &vec![3]));
let right = build_table(("a2", &vec![4]), ("b2", &vec![2]), ("c2", &vec![5]));
let left = build_table(
("a1", &vec![1, 2, 3]),
("b1", &vec![4, 5, 5]),
("c1", &vec![7, 8, 9]),
);
let right = build_table(
("a2", &vec![10, 20, 30]),
("b2", &vec![4, 5, 5]),
("c2", &vec![70, 8, 90]),
);
let on = vec![(
Arc::new(Column::new("b1", 1)) as _,
Arc::new(Column::new("b2", 1)) as _,
Expand Down Expand Up @@ -406,9 +414,9 @@ fn join_and_projection_for_pushdown(
Ok((join, projection))
}

#[test]
fn projection_pushdown_remaps_filter() -> Result<()> {
let filter = JoinFilter::new(
/// `c1 < c2`, referencing the last column of each child.
fn filter_for_pushdown() -> JoinFilter {
JoinFilter::new(
Arc::new(BinaryExpr::new(
Arc::new(Column::new("c1", 0)),
Operator::Lt,
Expand All @@ -428,8 +436,13 @@ fn projection_pushdown_remaps_filter() -> Result<()> {
Field::new("c1", DataType::Int32, false),
Field::new("c2", DataType::Int32, false),
])),
);
let (join, projection) = join_and_projection_for_pushdown(Some(filter))?;
)
}

#[test]
fn projection_pushdown_remaps_filter() -> Result<()> {
let (join, projection) =
join_and_projection_for_pushdown(Some(filter_for_pushdown()))?;

let swapped = join
.try_swapping_with_projection(&projection)?
Expand Down Expand Up @@ -458,6 +471,41 @@ fn projection_pushdown_remaps_filter() -> Result<()> {
Ok(())
}

#[tokio::test]
async fn projection_pushdown_remaps_filter_execute() -> Result<()> {
let (join, projection) =
join_and_projection_for_pushdown(Some(filter_for_pushdown()))?;
let expected_schema = projection.schema();

let swapped = join
.try_swapping_with_projection(&projection)?
.expect("projection should be pushed below the join");
assert!(
swapped.downcast_ref::<SortMergeJoinExec>().is_some(),
"projection should not be embedded in the join"
);
assert_eq!(swapped.schema(), expected_schema);

// The pushed-down children keep only `c*` and `b*`, so the filter must run
// against the remapped indices: `c1 < c2` filters out the (8, 8) and (9, 8)
// pairs of the `b1 = b2 = 5` group.
let batches =
common::collect(swapped.execute(0, Arc::new(TaskContext::default()))?).await?;

// The output order is important as SMJ preserves sortedness
assert_snapshot!(batches_to_string(&batches), @r"
+----+----+----+----+
| c1 | b1 | c2 | b2 |
+----+----+----+----+
| 7 | 4 | 70 | 4 |
| 8 | 5 | 90 | 5 |
| 9 | 5 | 90 | 5 |
+----+----+----+----+
");

Ok(())
}

#[test]
fn projection_pushdown_without_filter() -> Result<()> {
let (join, projection) = join_and_projection_for_pushdown(None)?;
Expand Down
30 changes: 30 additions & 0 deletions datafusion/sqllogictest/test_files/joins.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2857,6 +2857,36 @@ NULL 1970-01-04T00:00:00 789 ghi 1970-01-04 NULL 789 qwe
NULL NULL NULL NULL NULL 1970-01-04T00:00:00 0 qwerty
NULL NULL NULL NULL NULL NULL 100000 abcdefg

# Pushing the projection below the sort merge join must not drop a column that
# the join filter still needs (t2.c2 here). Instead the join applies the
# projection itself.
query TT
explain select t1.c1, t2.c1, t2.c3 from hashjoin_datatype_table_t1 t1 right join hashjoin_datatype_table_t2 t2 on t1.c1 = t2.c1 and t1.c2 < t2.c2
----
logical_plan
01)Projection: t1.c1, t2.c1, t2.c3
02)--Right Join: t1.c1 = t2.c1 Filter: t1.c2 < t2.c2
03)----SubqueryAlias: t1
04)------TableScan: hashjoin_datatype_table_t1 projection=[c1, c2]
05)----SubqueryAlias: t2
06)------TableScan: hashjoin_datatype_table_t2 projection=[c1, c2, c3]
physical_plan
01)SortMergeJoinExec: join_type=Right, on=[(c1@0, c1@0)], filter=c2@0 < c2@1, projection=[c1@0, c1@2, c3@4]
02)--SortExec: expr=[c1@0 ASC], preserve_partitioning=[true]
03)----RepartitionExec: partitioning=Hash([c1@0], 2), input_partitions=1
04)------DataSourceExec: partitions=1, partition_sizes=[1]
05)--SortExec: expr=[c1@0 ASC], preserve_partitioning=[true]
06)----RepartitionExec: partitioning=Hash([c1@0], 2), input_partitions=1
07)------DataSourceExec: partitions=1, partition_sizes=[1]

query DDR rowsort
select t1.c1, t2.c1, t2.c3 from hashjoin_datatype_table_t1 t1 right join hashjoin_datatype_table_t2 t2 on t1.c1 = t2.c1 and t1.c2 < t2.c2
----
NULL 1970-01-02 -123.12
NULL 1970-01-04 789
NULL NULL 0
NULL NULL 100000

# Regression test: projection optimization through SortMergeJoinExec must keep
# JoinFilter columns available when the output projection does not select them.
statement ok
Expand Down
18 changes: 9 additions & 9 deletions datafusion/sqllogictest/test_files/sort_merge_join_spill.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ EXPLAIN ANALYZE
SELECT p.k, w.v, length(w.p) FROM probe p JOIN wide w ON p.k = w.k
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Inner, on=[(k@0, k@0)], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Inner, on=[(k@0, k@0)], projection=[k@0, v@2, p@3], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe p JOIN wide w ON p.k = w.k
Expand All @@ -152,7 +152,7 @@ EXPLAIN ANALYZE
SELECT p.k, w.v, length(w.p) FROM probe p LEFT JOIN wide w ON p.k = w.k
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Left, on=[(k@0, k@0)], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Left, on=[(k@0, k@0)], projection=[k@0, v@2, p@3], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe p LEFT JOIN wide w ON p.k = w.k
Expand All @@ -164,7 +164,7 @@ EXPLAIN ANALYZE
SELECT p.k, w.v, length(w.p) FROM wide w RIGHT JOIN probe p ON p.k = w.k
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Right, on=[(k@0, k@0)], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Right, on=[(k@0, k@0)], projection=[k@3, v@1, p@2], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM wide w RIGHT JOIN probe p ON p.k = w.k
Expand All @@ -176,7 +176,7 @@ EXPLAIN ANALYZE
SELECT p.k, w.v, length(w.p) FROM probe p FULL JOIN wide w ON p.k = w.k
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], projection=[k@0, v@2, p@3], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe p FULL JOIN wide w ON p.k = w.k
Expand All @@ -192,7 +192,7 @@ SELECT p.k, w.v, length(w.p) FROM probe p
JOIN wide w ON p.k = w.k AND p.x < w.x
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Inner, on=[(k@0, k@0)], filter=x@0 < x@1, metrics=[output_rows=900,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Inner, on=[(k@0, k@0)], filter=x@0 < x@1, projection=[k@0, v@3, p@5], metrics=[output_rows=900,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe p
Expand All @@ -206,7 +206,7 @@ SELECT p.k, w.v, length(w.p) FROM probe p
LEFT JOIN wide w ON p.k = w.k AND p.x < w.x
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Left, on=[(k@0, k@0)], filter=x@0 < x@1, metrics=[output_rows=902,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Left, on=[(k@0, k@0)], filter=x@0 < x@1, projection=[k@0, v@3, p@5], metrics=[output_rows=902,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe p
Expand All @@ -220,7 +220,7 @@ SELECT p.k, w.v, length(w.p) FROM wide w
RIGHT JOIN probe p ON p.k = w.k AND p.x < w.x
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Right, on=[(k@0, k@0)], filter=x@1 < x@0, metrics=[output_rows=902,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Right, on=[(k@0, k@0)], filter=x@1 < x@0, projection=[k@4, v@1, p@3], metrics=[output_rows=902,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM wide w
Expand All @@ -234,7 +234,7 @@ SELECT p.k, w.v, length(w.p) FROM probe p
FULL JOIN wide w ON p.k = w.k AND p.x < w.x
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], filter=x@0 < x@1, metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], filter=x@0 < x@1, projection=[k@0, v@3, p@5], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe p
Expand All @@ -249,7 +249,7 @@ EXPLAIN ANALYZE
SELECT p.k, w.v, length(w.p) FROM probe_nomatch p FULL JOIN wide w ON p.k = w.k
----
Plan with Metrics
<slt:ignore>SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>
<slt:ignore>SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], projection=[k@0, v@2, p@3], metrics=[output_rows=2.00 K,<slt:ignore>spill_count=10, spilled_bytes=<slt:ignore>spilled_rows=2.00 K, peak_mem_used=<slt:ignore>

query III rowsort
SELECT p.k, w.v, length(w.p) FROM probe_nomatch p FULL JOIN wide w ON p.k = w.k
Expand Down