Skip to content

[fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException#60914

Open
morrySnow wants to merge 1 commit intoapache:masterfrom
morrySnow:iow-bug
Open

[fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException#60914
morrySnow wants to merge 1 commit intoapache:masterfrom
morrySnow:iow-bug

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Feb 28, 2026 •
edited
Loading

What problem does this PR solve?

Problem Summary:

When performing insert overwrite on a whole table, partition names are snapshotted at planning time without holding the table lock. A concurrent DROP PARTITION between the snapshot and the replace step causes Env.replaceTempPartition() to throw DdlException because the original partition no longer exists.

Fix: after acquiring the table write lock in InsertOverwriteUtil.replacePartition(), filter out any original partitions that have been deleted before constructing ReplacePartitionOp. The temp partition list is kept intact so all written data remains visible.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@morrySnow
[fix](insert-overwrite) skip deleted partitions before replace to avo…
160ee55 
…id DdlException

When performing insert overwrite on a whole table, partition names are
snapshotted at planning time without holding the table lock. A concurrent
DROP PARTITION between the snapshot and the replace step causes
Env.replaceTempPartition() to throw DdlException because the original
partition no longer exists.

Fix: after acquiring the table write lock in InsertOverwriteUtil.replacePartition(),
filter out any original partitions that have been deleted before constructing
ReplacePartitionOp. The temp partition list is kept intact so all written
data remains visible.
@morrySnow
Copy link
Contributor Author

morrySnow commented Feb 28, 2026

run buildall

@Thearas
Copy link
Contributor

Thearas commented Feb 28, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Feb 28, 2026

TPC-H: Total hot run time: 28900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 160ee55af10a7aaff412ee4c81e0034a9cca2dd2, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17603	4465	4309	4309
q2	q3	10654	800	524	524
q4	4677	356	267	267
q5	7547	1193	1054	1054
q6	178	176	152	152
q7	793	831	672	672
q8	9292	1488	1328	1328
q9	4864	4780	4762	4762
q10	6774	1886	1638	1638
q11	456	241	238	238
q12	682	568	461	461
q13	17757	4226	3424	3424
q14	223	228	210	210
q15	924	801	793	793
q16	709	729	668	668
q17	704	870	425	425
q18	6013	5395	5263	5263
q19	1236	990	646	646
q20	515	494	393	393
q21	4701	1836	1432	1432
q22	336	289	241	241
Total cold run time: 96638 ms
Total hot run time: 28900 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4409	4322	4351	4322
q2	q3	1762	2203	1738	1738
q4	881	1160	778	778
q5	4032	4316	4333	4316
q6	185	173	143	143
q7	1706	1610	1473	1473
q8	2419	2663	2520	2520
q9	7544	7496	7467	7467
q10	2707	2878	2448	2448
q11	520	442	407	407
q12	489	582	447	447
q13	3988	4453	3737	3737
q14	301	324	301	301
q15	877	831	828	828
q16	742	750	700	700
q17	1155	1518	1330	1330
q18	7145	6748	6655	6655
q19	966	982	930	930
q20	2060	2134	1973	1973
q21	3984	3513	3337	3337
q22	481	444	380	380
Total cold run time: 48353 ms
Total hot run time: 46230 ms

@doris-robot
Copy link

doris-robot commented Feb 28, 2026

TPC-DS: Total hot run time: 184762 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 160ee55af10a7aaff412ee4c81e0034a9cca2dd2, data reload: false

query5	5240	641	509	509
query6	320	237	229	229
query7	4227	480	267	267
query8	355	253	246	246
query9	8750	2746	2729	2729
query10	555	392	341	341
query11	17002	17577	17083	17083
query12	199	135	134	134
query13	1338	501	371	371
query14	7776	3437	3141	3141
query14_1	2999	2917	2847	2847
query15	207	198	175	175
query16	1007	476	488	476
query17	1308	815	651	651
query18	2892	470	384	384
query19	228	268	274	268
query20	157	133	138	133
query21	232	145	145	145
query22	5645	5819	5633	5633
query23	17404	17149	16997	16997
query23_1	16676	16635	16639	16635
query24	7025	1607	1204	1204
query24_1	1192	1216	1205	1205
query25	530	448	402	402
query26	1232	258	144	144
query27	2747	458	285	285
query28	4499	1858	1856	1856
query29	795	558	472	472
query30	309	244	209	209
query31	871	730	652	652
query32	76	72	65	65
query33	506	340	270	270
query34	894	909	608	608
query35	628	674	609	609
query36	1104	1169	985	985
query37	133	99	82	82
query38	2971	2943	2859	2859
query39	896	865	827	827
query39_1	830	823	832	823
query40	225	150	138	138
query41	66	58	56	56
query42	109	102	99	99
query43	377	375	353	353
query44	
query45	194	182	183	182
query46	949	980	602	602
query47	2087	2128	2046	2046
query48	303	315	232	232
query49	628	448	383	383
query50	687	281	209	209
query51	4049	4106	4054	4054
query52	105	107	99	99
query53	287	341	285	285
query54	291	263	251	251
query55	92	80	80	80
query56	312	306	305	305
query57	1367	1335	1270	1270
query58	286	272	273	272
query59	2612	2721	2538	2538
query60	335	337	315	315
query61	148	148	149	148
query62	607	600	533	533
query63	311	279	274	274
query64	4822	1252	994	994
query65	
query66	1366	482	365	365
query67	16403	16478	16194	16194
query68	
query69	388	306	280	280
query70	983	982	939	939
query71	332	308	297	297
query72	2778	2660	2421	2421
query73	539	543	323	323
query74	9971	9945	9719	9719
query75	2837	2746	2467	2467
query76	2294	1042	657	657
query77	365	392	310	310
query78	11251	11368	10721	10721
query79	1117	796	599	599
query80	876	625	543	543
query81	526	275	245	245
query82	1342	151	114	114
query83	346	260	238	238
query84	255	120	99	99
query85	924	479	453	453
query86	410	317	293	293
query87	3109	3137	3055	3055
query88	3535	2666	2649	2649
query89	441	369	345	345
query90	1799	172	169	169
query91	169	162	134	134
query92	78	73	72	72
query93	911	812	511	511
query94	504	325	301	301
query95	580	394	317	317
query96	636	514	230	230
query97	2513	2483	2417	2417
query98	237	224	226	224
query99	991	994	903	903
Total cold run time: 255092 ms
Total hot run time: 184762 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

dev/4.0.x

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@morrySnow @Thearas @doris-robot