Skip to content

[feat](condition cache) Support condition cache for external table#60897

Open
jacktengg wants to merge 1 commit intoapache:masterfrom
jacktengg:condition-cache-cc
Open

[feat](condition cache) Support condition cache for external table#60897
jacktengg wants to merge 1 commit intoapache:masterfrom
jacktengg:condition-cache-cc

Conversation

@jacktengg
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 27, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28620 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f140c7df2398a491f977da25906a10cce20fee5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17630	4532	4309	4309
q2	q3	10648	780	520	520
q4	4678	348	260	260
q5	7547	1194	1014	1014
q6	177	173	146	146
q7	766	834	666	666
q8	9287	1479	1350	1350
q9	4867	4715	4653	4653
q10	7155	1891	1627	1627
q11	474	252	239	239
q12	705	565	469	469
q13	17817	4230	3445	3445
q14	229	231	223	223
q15	960	801	798	798
q16	751	726	670	670
q17	698	845	443	443
q18	5980	5338	5173	5173
q19	1201	972	610	610
q20	511	482	389	389
q21	4622	1900	1375	1375
q22	339	282	241	241
Total cold run time: 97042 ms
Total hot run time: 28620 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4444	4366	4359	4359
q2	q3	1759	2169	1726	1726
q4	848	1158	758	758
q5	4049	4298	4330	4298
q6	183	177	142	142
q7	1730	1606	1504	1504
q8	2423	2669	2541	2541
q9	7395	7464	7349	7349
q10	2630	2914	2453	2453
q11	526	438	416	416
q12	505	578	446	446
q13	3928	4528	3739	3739
q14	291	307	282	282
q15	848	809	790	790
q16	731	798	720	720
q17	1167	1508	1367	1367
q18	7086	6795	6584	6584
q19	857	982	912	912
q20	2097	2129	2048	2048
q21	4013	3639	3429	3429
q22	456	456	392	392
Total cold run time: 47966 ms
Total hot run time: 46255 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184197 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f140c7df2398a491f977da25906a10cce20fee5, data reload: false

query5	5159	638	530	530
query6	338	231	209	209
query7	4233	481	264	264
query8	329	240	227	227
query9	8726	2744	2730	2730
query10	560	388	315	315
query11	16928	17594	17394	17394
query12	202	132	131	131
query13	1430	498	359	359
query14	7049	3234	3018	3018
query14_1	2881	2828	3036	2828
query15	205	200	186	186
query16	1152	558	460	460
query17	2154	720	612	612
query18	2857	513	393	393
query19	207	215	193	193
query20	137	124	126	124
query21	214	137	119	119
query22	5860	5635	5110	5110
query23	17167	16772	16582	16582
query23_1	16696	16629	16626	16626
query24	7071	1587	1198	1198
query24_1	1210	1257	1226	1226
query25	537	451	395	395
query26	1226	261	144	144
query27	2795	473	281	281
query28	4513	1860	1857	1857
query29	787	545	465	465
query30	308	246	205	205
query31	854	708	654	654
query32	81	68	68	68
query33	508	340	275	275
query34	898	903	588	588
query35	618	670	621	621
query36	1092	1101	1020	1020
query37	134	95	83	83
query38	2914	2889	2832	2832
query39	896	867	840	840
query39_1	832	830	834	830
query40	226	155	133	133
query41	62	61	58	58
query42	109	99	103	99
query43	370	381	341	341
query44	
query45	196	191	180	180
query46	877	995	599	599
query47	2113	2143	2029	2029
query48	305	303	225	225
query49	624	463	372	372
query50	677	283	213	213
query51	4152	4049	4077	4049
query52	107	110	97	97
query53	287	333	285	285
query54	299	258	253	253
query55	88	88	90	88
query56	320	306	305	305
query57	1358	1344	1295	1295
query58	303	272	299	272
query59	2553	2770	2606	2606
query60	331	336	318	318
query61	150	145	148	145
query62	611	601	533	533
query63	306	277	274	274
query64	4841	1351	1081	1081
query65	
query66	1389	467	379	379
query67	16235	16471	16453	16453
query68	
query69	401	330	292	292
query70	991	928	959	928
query71	343	310	301	301
query72	3012	2814	2530	2530
query73	563	542	314	314
query74	9939	9886	9749	9749
query75	2824	2750	2454	2454
query76	2302	1030	664	664
query77	356	376	303	303
query78	11167	11319	10690	10690
query79	3020	804	598	598
query80	1763	614	535	535
query81	590	288	243	243
query82	992	150	115	115
query83	340	257	246	246
query84	251	113	99	99
query85	917	473	422	422
query86	519	322	296	296
query87	3105	3061	3088	3061
query88	3507	2640	2632	2632
query89	423	366	337	337
query90	1936	171	167	167
query91	155	146	132	132
query92	79	76	69	69
query93	1425	814	519	519
query94	641	303	299	299
query95	577	388	310	310
query96	630	517	230	230
query97	2463	2531	2411	2411
query98	235	213	209	209
query99	996	970	911	911
Total cold run time: 258686 ms
Total hot run time: 184197 ms

@jacktengg
Copy link
Contributor Author

run p0

@jacktengg
Copy link
Contributor Author

run external

@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28745 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b882d226e94fc72abb8bf9704a41541423407c4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17635	4548	4354	4354
q2	q3	10647	792	521	521
q4	4680	352	257	257
q5	7536	1204	1025	1025
q6	174	177	148	148
q7	783	858	676	676
q8	9308	1448	1307	1307
q9	4863	4727	4712	4712
q10	6817	1876	1622	1622
q11	454	256	246	246
q12	695	563	465	465
q13	17772	4203	3443	3443
q14	228	228	206	206
q15	967	793	783	783
q16	754	716	671	671
q17	710	852	411	411
q18	6017	5397	5263	5263
q19	1113	985	613	613
q20	491	496	387	387
q21	4308	1854	1400	1400
q22	335	279	235	235
Total cold run time: 96287 ms
Total hot run time: 28745 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4437	4362	4361	4361
q2	q3	1769	2177	1732	1732
q4	849	1156	750	750
q5	4018	4330	4316	4316
q6	178	177	141	141
q7	1723	1599	1476	1476
q8	2439	2653	2538	2538
q9	7332	7571	7417	7417
q10	2693	2809	2453	2453
q11	505	442	414	414
q12	503	600	468	468
q13	3932	4426	3702	3702
q14	305	312	290	290
q15	867	812	843	812
q16	733	772	710	710
q17	1167	1526	1329	1329
q18	7220	6885	6749	6749
q19	925	857	887	857
q20	2121	2190	2042	2042
q21	3912	3675	3380	3380
q22	489	429	382	382
Total cold run time: 48117 ms
Total hot run time: 46319 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184276 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b882d226e94fc72abb8bf9704a41541423407c4, data reload: false

query5	5055	644	524	524
query6	325	220	196	196
query7	4212	474	268	268
query8	342	244	224	224
query9	8737	2723	2705	2705
query10	570	388	344	344
query11	16940	16810	16608	16608
query12	191	131	124	124
query13	1282	452	355	355
query14	6516	3247	2942	2942
query14_1	2858	2827	2845	2827
query15	213	196	185	185
query16	990	457	441	441
query17	1041	689	572	572
query18	2588	425	335	335
query19	200	203	176	176
query20	138	127	126	126
query21	219	144	120	120
query22	5020	5784	5503	5503
query23	17583	17244	17155	17155
query23_1	17034	17001	17002	17001
query24	7378	1659	1230	1230
query24_1	1237	1259	1239	1239
query25	590	492	425	425
query26	1235	262	159	159
query27	2759	475	297	297
query28	4496	1866	1864	1864
query29	833	575	489	489
query30	308	247	208	208
query31	862	725	656	656
query32	81	72	75	72
query33	536	346	297	297
query34	936	900	568	568
query35	653	673	607	607
query36	1117	1163	946	946
query37	151	99	83	83
query38	2919	2881	2880	2880
query39	1035	871	863	863
query39_1	839	828	833	828
query40	234	158	138	138
query41	67	64	63	63
query42	112	105	105	105
query43	376	387	356	356
query44	
query45	200	189	182	182
query46	864	996	603	603
query47	2160	2127	2049	2049
query48	311	316	234	234
query49	645	476	378	378
query50	682	280	225	225
query51	4124	4141	4013	4013
query52	106	108	99	99
query53	305	335	303	303
query54	327	314	262	262
query55	85	85	80	80
query56	313	304	319	304
query57	1376	1329	1272	1272
query58	290	280	271	271
query59	2558	2599	2557	2557
query60	344	341	324	324
query61	150	143	149	143
query62	627	595	551	551
query63	313	269	278	269
query64	4904	1271	984	984
query65	
query66	1408	457	371	371
query67	16378	16366	16244	16244
query68	
query69	400	297	280	280
query70	976	1003	985	985
query71	339	304	299	299
query72	2789	2683	2432	2432
query73	539	540	320	320
query74	9977	9860	9761	9761
query75	2830	2741	2459	2459
query76	2313	1030	668	668
query77	355	388	318	318
query78	11193	11360	10692	10692
query79	1134	795	599	599
query80	1358	644	544	544
query81	553	276	252	252
query82	1014	159	119	119
query83	342	259	238	238
query84	250	119	97	97
query85	909	480	441	441
query86	417	320	278	278
query87	3111	3084	2943	2943
query88	3526	2663	2662	2662
query89	418	376	347	347
query90	1971	172	165	165
query91	169	156	135	135
query92	78	78	73	73
query93	973	817	509	509
query94	639	327	298	298
query95	581	396	315	315
query96	631	515	224	224
query97	2471	2511	2438	2438
query98	231	211	213	211
query99	1002	1033	909	909
Total cold run time: 254353 ms
Total hot run time: 184276 ms

@jacktengg jacktengg force-pushed the condition-cache-cc branch 4 times, most recently from 536faf2 to 6781f14 Compare February 28, 2026 10:59
@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 29158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15fff1b61996813f8ea89e9a4a8b70b3bfd7b798, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17612	4540	4303	4303
q2	q3	10648	775	526	526
q4	4681	363	268	268
q5	7544	1215	1016	1016
q6	173	176	148	148
q7	789	876	686	686
q8	9306	1511	1333	1333
q9	4964	4715	4718	4715
q10	6841	1886	1659	1659
q11	455	269	250	250
q12	696	574	466	466
q13	17782	4269	3459	3459
q14	236	241	220	220
q15	936	802	788	788
q16	775	727	680	680
q17	751	868	440	440
q18	6061	5438	5299	5299
q19	1255	987	652	652
q20	512	500	396	396
q21	5002	2001	1581	1581
q22	395	324	273	273
Total cold run time: 97414 ms
Total hot run time: 29158 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4688	4538	4574	4538
q2	q3	1798	2238	1771	1771
q4	869	1210	784	784
q5	4044	4417	4300	4300
q6	189	175	141	141
q7	1775	1647	1505	1505
q8	2517	2897	2555	2555
q9	7482	7351	7321	7321
q10	2597	2789	2398	2398
q11	516	436	411	411
q12	511	585	465	465
q13	4026	4481	3685	3685
q14	287	299	277	277
q15	904	805	793	793
q16	696	758	724	724
q17	1220	1473	1379	1379
q18	7271	6746	6597	6597
q19	922	915	913	913
q20	2191	2127	2024	2024
q21	3994	3504	3394	3394
q22	477	433	367	367
Total cold run time: 48974 ms
Total hot run time: 46342 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183955 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15fff1b61996813f8ea89e9a4a8b70b3bfd7b798, data reload: false

query5	4560	647	532	532
query6	330	219	215	215
query7	4222	474	273	273
query8	362	255	242	242
query9	8778	2728	2740	2728
query10	499	391	352	352
query11	17065	17602	17287	17287
query12	202	142	130	130
query13	1334	508	381	381
query14	7164	3360	2997	2997
query14_1	2929	2893	2964	2893
query15	247	214	182	182
query16	1015	473	472	472
query17	1119	744	647	647
query18	2709	429	342	342
query19	197	202	176	176
query20	138	125	127	125
query21	214	131	110	110
query22	4734	4849	4916	4849
query23	17253	16723	16566	16566
query23_1	16644	16662	16545	16545
query24	7134	1625	1222	1222
query24_1	1222	1198	1228	1198
query25	539	452	407	407
query26	1239	293	149	149
query27	2753	482	283	283
query28	4522	1855	1871	1855
query29	784	567	464	464
query30	310	244	210	210
query31	883	738	643	643
query32	77	73	69	69
query33	512	348	274	274
query34	905	893	560	560
query35	630	674	599	599
query36	1086	1124	997	997
query37	136	94	82	82
query38	2971	2937	2872	2872
query39	881	874	831	831
query39_1	826	823	828	823
query40	230	156	139	139
query41	63	60	60	60
query42	109	108	105	105
query43	387	387	361	361
query44	
query45	230	189	178	178
query46	871	988	601	601
query47	2105	2147	2033	2033
query48	305	315	243	243
query49	627	468	377	377
query50	686	278	212	212
query51	4083	4107	4100	4100
query52	105	106	98	98
query53	291	339	288	288
query54	293	267	259	259
query55	86	87	77	77
query56	325	303	302	302
query57	1363	1357	1279	1279
query58	288	275	264	264
query59	2617	2718	2595	2595
query60	330	326	333	326
query61	149	146	143	143
query62	622	597	529	529
query63	315	276	281	276
query64	4863	1282	995	995
query65	
query66	1400	455	360	360
query67	16484	16325	16236	16236
query68	
query69	397	305	297	297
query70	958	963	982	963
query71	334	307	301	301
query72	2991	2825	2568	2568
query73	548	555	336	336
query74	9975	9936	9717	9717
query75	2859	2776	2472	2472
query76	2288	1046	692	692
query77	393	426	319	319
query78	11325	11574	10860	10860
query79	1775	784	598	598
query80	1365	626	543	543
query81	572	283	252	252
query82	998	146	113	113
query83	366	261	240	240
query84	256	133	99	99
query85	1050	501	427	427
query86	408	331	329	329
query87	3140	3081	3022	3022
query88	3569	2681	2681	2681
query89	436	372	345	345
query90	1884	170	176	170
query91	166	157	137	137
query92	83	80	67	67
query93	1077	877	514	514
query94	636	333	309	309
query95	589	401	310	310
query96	634	517	229	229
query97	2458	2495	2402	2402
query98	231	221	220	220
query99	1004	1002	922	922
Total cold run time: 254763 ms
Total hot run time: 183955 ms

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 35.69% (121/339) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.63% (19663/37359)
Line Coverage 36.24% (183623/506679)
Region Coverage 32.54% (142483/437833)
Branch Coverage 33.46% (61743/184537)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 64.31% (218/339) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.47% (26146/36581)
Line Coverage 54.24% (273986/505126)
Region Coverage 51.63% (228177/441970)
Branch Coverage 52.84% (97811/185101)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants