Skip to content

[fix](nereids) Bound outer join null-reject inference to nullable outputs#65250

Open
foxtail463 wants to merge 2 commits into
apache:masterfrom
foxtail463:fix/outer-join-null-reject
Open

[fix](nereids) Bound outer join null-reject inference to nullable outputs#65250
foxtail463 wants to merge 2 commits into
apache:masterfrom
foxtail463:fix/outer-join-null-reject

Conversation

@foxtail463

Copy link
Copy Markdown
Contributor

Problem Summary:
Outer join elimination uses fold-based null-reject inference to decide whether
nullable-side rows from the current outer join can be filtered away. This check
only needs slots from that join's nullable-side outputs, but the previous flow
could also test unrelated predicate inputs. When a filter predicate references a
mark slot produced by another join, folding that slot is useless for eliminating
the current outer join and can be costly when the fixed-point rewrite batch
revisits the same predicate multiple times.

Solution:
Add an API to infer null-rejecting slots only for a given target slot set, and
use the current join's nullable-side outputs as that target in
EliminateOuterJoin. Skip mark-join slots as inference targets, preserve the
existing expression complexity limits, and avoid rewriting when the join type
does not change.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29409 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4fabf702a8672134d82ea4f08d9bd69a59f3d22, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17654	4032	4011	4011
q2	2047	326	201	201
q3	10238	1390	852	852
q4	4715	475	340	340
q5	7742	850	584	584
q6	244	168	140	140
q7	774	831	630	630
q8	10030	1644	1562	1562
q9	6049	4495	4483	4483
q10	6821	1820	1509	1509
q11	498	350	320	320
q12	704	556	449	449
q13	18104	3369	2795	2795
q14	274	261	244	244
q15	q16	792	789	720	720
q17	1744	1165	757	757
q18	7072	5806	5491	5491
q19	2777	1318	967	967
q20	795	686	579	579
q21	5962	2749	2476	2476
q22	444	356	299	299
Total cold run time: 105480 ms
Total hot run time: 29409 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4408	4244	4239	4239
q2	281	325	228	228
q3	4552	4950	4417	4417
q4	2095	2138	1383	1383
q5	4425	4291	4627	4291
q6	262	198	160	160
q7	2048	1858	1680	1680
q8	2509	2147	2120	2120
q9	7909	7677	7791	7677
q10	4746	4689	4396	4396
q11	572	423	396	396
q12	772	759	542	542
q13	3317	3571	2980	2980
q14	287	318	296	296
q15	q16	714	757	656	656
q17	1339	1330	1340	1330
q18	7796	7504	6664	6664
q19	1091	1091	1072	1072
q20	2276	2213	1933	1933
q21	5196	4602	4496	4496
q22	508	453	427	427
Total cold run time: 57103 ms
Total hot run time: 51383 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 90.24% (37/41) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 173640 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4fabf702a8672134d82ea4f08d9bd69a59f3d22, data reload: false

query5	4448	633	485	485
query6	474	217	205	205
query7	4844	582	369	369
query8	335	186	173	173
query9	8718	4064	4088	4064
query10	507	346	300	300
query11	5950	2335	2199	2199
query12	158	102	104	102
query13	1256	620	476	476
query14	6299	5287	4985	4985
query14_1	4300	4278	4301	4278
query15	217	201	179	179
query16	1033	463	452	452
query17	1140	716	584	584
query18	2554	496	354	354
query19	219	195	154	154
query20	112	108	109	108
query21	236	167	135	135
query22	13600	13567	13394	13394
query23	17377	16464	16183	16183
query23_1	16283	16364	16231	16231
query24	7656	1761	1323	1323
query24_1	1330	1298	1312	1298
query25	571	459	392	392
query26	1331	358	210	210
query27	2588	559	390	390
query28	4408	2039	2032	2032
query29	1096	602	526	526
query30	333	268	228	228
query31	1131	1096	968	968
query32	114	65	62	62
query33	563	323	262	262
query34	1220	1126	645	645
query35	781	794	682	682
query36	1407	1371	1283	1283
query37	160	107	96	96
query38	1883	1705	1654	1654
query39	926	947	894	894
query39_1	901	884	895	884
query40	259	199	135	135
query41	64	62	62	62
query42	93	94	99	94
query43	318	323	283	283
query44	1425	800	791	791
query45	198	182	176	176
query46	1041	1236	722	722
query47	2336	2302	2267	2267
query48	384	434	294	294
query49	580	421	317	317
query50	1106	440	321	321
query51	4399	4350	4269	4269
query52	82	83	74	74
query53	259	265	207	207
query54	271	230	210	210
query55	75	72	65	65
query56	285	288	284	284
query57	1393	1410	1322	1322
query58	284	256	258	256
query59	1542	1592	1418	1418
query60	308	266	231	231
query61	153	146	170	146
query62	696	648	586	586
query63	234	206	209	206
query64	2531	764	598	598
query65	4896	4786	4747	4747
query66	1816	522	374	374
query67	29507	29406	29361	29361
query68	3141	1520	1045	1045
query69	412	309	272	272
query70	1041	966	915	915
query71	355	329	292	292
query72	2935	2234	2303	2234
query73	911	806	421	421
query74	5123	4968	4811	4811
query75	2614	2571	2218	2218
query76	2330	1183	803	803
query77	359	377	278	278
query78	12516	12516	11802	11802
query79	1430	1219	780	780
query80	1277	542	467	467
query81	536	326	288	288
query82	630	156	120	120
query83	375	312	292	292
query84	284	163	130	130
query85	981	592	517	517
query86	478	309	265	265
query87	1834	1822	1755	1755
query88	3743	2801	2810	2801
query89	465	398	363	363
query90	1872	198	194	194
query91	198	189	162	162
query92	66	64	54	54
query93	1785	1536	980	980
query94	776	349	303	303
query95	771	482	543	482
query96	1027	796	365	365
query97	2686	2689	2597	2597
query98	214	210	201	201
query99	1157	1161	1022	1022
Total cold run time: 259651 ms
Total hot run time: 173640 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4fabf702a8672134d82ea4f08d9bd69a59f3d22, data reload: false

query1	0.00	0.01	0.00
query2	0.10	0.05	0.05
query3	0.26	0.13	0.14
query4	1.60	0.14	0.14
query5	0.24	0.23	0.21
query6	1.26	1.04	1.11
query7	0.04	0.00	0.01
query8	0.06	0.04	0.04
query9	0.40	0.32	0.32
query10	0.53	0.54	0.56
query11	0.19	0.14	0.15
query12	0.17	0.14	0.14
query13	0.48	0.46	0.49
query14	1.01	0.99	1.01
query15	0.61	0.60	0.59
query16	0.32	0.32	0.32
query17	1.08	1.13	1.14
query18	0.23	0.21	0.21
query19	2.04	1.97	2.02
query20	0.02	0.02	0.01
query21	15.45	0.21	0.14
query22	4.84	0.05	0.05
query23	16.13	0.32	0.12
query24	2.98	0.42	0.36
query25	0.15	0.05	0.06
query26	0.72	0.20	0.15
query27	0.03	0.04	0.03
query28	3.53	0.90	0.52
query29	12.49	4.41	3.54
query30	0.28	0.16	0.15
query31	2.77	0.59	0.31
query32	3.22	0.61	0.50
query33	3.17	3.22	3.27
query34	15.65	4.23	3.54
query35	3.54	3.54	3.55
query36	0.56	0.44	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.68 s
Total hot run time: 25.41 s

@morrySnow morrySnow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR! The bounding of null-reject inference to nullable-side outputs is a clean optimization. I reviewed all angles (line-by-line, removed-behavior audit, cross-file tracer, reuse, simplification, efficiency, altitude, conventions) and found the implementation correct and well-tested.

One code organization observation (not a bug): The new getNullableSideOutput() in LogicalJoin re-encodes the join-type-to-nullable-side mapping (LEFT_OUTER→right nullable, RIGHT_OUTER→left nullable, FULL_OUTER→both) that already exists in JoinUtils.getJoinOutput() lines 417-433. If a new join type with different null-extension semantics is added, the mapping would need updating in multiple places. Consider whether a utility method on JoinType (e.g., isLeftSideNullable()/isRightSideNullable()) could consolidate this knowledge.

LGTM overall — the new test coverage is thorough (IN+OR-FALSE elimination, FULL OUTER degradation, early-return guard, negative cases).

Comment thread fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java Outdated
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 53.73% (36/67) 🎉
Increment coverage report
Complete coverage report

@yx-keith yx-keith left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bounding the inference to getNullableSideOutput() is the right target set. Note this isn't purely skipping useless work — it changes elimination behavior: skipping mark-join slots as inference targets can drop an elimination, and the smaller target set interacts with the expression-complexity limit (a set that previously bailed on complexity may now fit and eliminate more).

P0 Regression / cloud_p0 are red. Since outer-join elimination affects results, please confirm these are expected plan changes (regenerated .out) rather than wrong results, and that skipping mark-join slots is always safe for eliminating the current join.

@foxtail463 foxtail463 requested a review from zclllyybb as a code owner July 7, 2026 12:30
@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

… and JoinUtils. Also remove the redundant mark-slot guard from target-based not-null inference
@foxtail463 foxtail463 force-pushed the fix/outer-join-null-reject branch from 1722a9b to 1e9ee9c Compare July 7, 2026 13:14
@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants