Skip to content

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Jun 10, 2025

What problem does this PR solve?

When export parallelism exceeds maximum_parallelism_of_export_job, excess tasks are discarded by LogDiscardPolicy.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

  • Replace newDaemonCacheThreadPool with newDaemonFixedThreadPool to use blocking queue instead of discard policy
  • Dynamically calculate queue size as max(parallelNum, maximum_parallelism_of_export_job) to ensure all tasks can be queued
  • Move thread pool creation from constructor to exec() method for dynamic sizing

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk zy-kkk changed the title The export task thread pool uses a blocking queue strategy [fix](export) The export task thread pool uses a blocking queue strategy Jun 10, 2025
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 11, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49676 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef568041bb4cd58cce2e0752fb1951ce0c59bc83, data reload: false

------ Round 1 ----------------------------------
q1	17805	4424	4440	4424
q2	2098	161	155	155
q3	10255	1923	1961	1923
q4	10151	1295	1318	1295
q5	8544	3951	3976	3951
q6	238	126	126	126
q7	2042	1656	1632	1632
q8	9318	2759	2721	2721
q9	10451	10280	9980	9980
q10	8687	3552	3544	3544
q11	422	246	251	246
q12	478	308	303	303
q13	18383	3955	4065	3955
q14	359	327	328	327
q15	506	462	461	461
q16	544	467	469	467
q17	1144	952	958	952
q18	7307	6902	6890	6890
q19	1668	1564	1540	1540
q20	514	317	304	304
q21	4394	4088	4086	4086
q22	505	394	405	394
Total cold run time: 115813 ms
Total hot run time: 49676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4328	4327	4304	4304
q2	329	229	226	226
q3	4159	4145	4143	4143
q4	2748	2747	2751	2747
q5	7229	7188	7143	7143
q6	240	120	119	119
q7	3275	2834	2909	2834
q8	4373	4451	4520	4451
q9	14178	14182	13872	13872
q10	4231	4279	4305	4279
q11	777	699	681	681
q12	1032	835	851	835
q13	6982	3753	3742	3742
q14	452	441	420	420
q15	504	452	460	452
q16	629	606	602	602
q17	3819	3856	3941	3856
q18	8789	8691	8677	8677
q19	1726	1638	1644	1638
q20	2361	2132	2117	2117
q21	8348	8445	8482	8445
q22	990	978	944	944
Total cold run time: 81499 ms
Total hot run time: 76527 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 213227 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef568041bb4cd58cce2e0752fb1951ce0c59bc83, data reload: false

query1	952	401	417	401
query2	6555	2283	2145	2145
query3	6928	204	203	203
query4	23582	22115	21405	21405
query5	19722	6543	6528	6528
query6	288	220	232	220
query7	4342	300	322	300
query8	275	226	270	226
query9	3033	2634	2571	2571
query10	471	310	318	310
query11	15965	15012	15251	15012
query12	132	77	79	77
query13	1038	454	440	440
query14	16855	13389	13777	13389
query15	364	221	228	221
query16	6476	285	264	264
query17	1740	942	918	918
query18	897	317	318	317
query19	220	152	150	150
query20	105	91	95	91
query21	195	101	90	90
query22	5200	5062	5137	5062
query23	34450	33675	33516	33516
query24	7920	6335	6323	6323
query25	539	431	431	431
query26	1274	162	165	162
query27	2453	298	291	291
query28	6110	2289	2255	2255
query29	3003	2780	2781	2780
query30	243	168	169	168
query31	967	736	752	736
query32	72	63	61	61
query33	459	275	278	275
query34	857	491	471	471
query35	1170	1010	966	966
query36	1356	1185	1457	1185
query37	174	60	61	60
query38	3102	2968	2910	2910
query39	1370	1321	1322	1321
query40	314	98	97	97
query41	41	39	37	37
query42	92	82	82	82
query43	624	643	610	610
query44	1140	723	720	720
query45	255	227	236	227
query46	1233	973	957	957
query47	1954	1863	1875	1863
query48	516	409	405	405
query49	664	376	404	376
query50	866	616	646	616
query51	4764	4704	4712	4704
query52	102	81	87	81
query53	224	183	191	183
query54	2682	2456	2502	2456
query55	85	89	83	83
query56	230	196	210	196
query57	1282	1151	1200	1151
query58	225	216	214	214
query59	3542	3445	3250	3250
query60	211	208	205	205
query61	97	95	98	95
query62	704	486	525	486
query63	203	186	179	179
query64	3565	1594	1470	1470
query65	3689	3562	3574	3562
query66	771	433	448	433
query67	17532	16189	15264	15264
query68	9212	633	649	633
query69	509	277	271	271
query70	1504	1408	1328	1328
query71	404	308	311	308
query72	6961	4771	4820	4771
query73	764	317	335	317
query74	6326	5881	5808	5808
query75	4667	3738	3693	3693
query76	4732	1125	1212	1125
query77	638	251	257	251
query78	12735	11819	11710	11710
query79	7889	641	663	641
query80	2558	404	381	381
query81	514	235	243	235
query82	1640	102	98	98
query83	168	136	134	134
query84	270	75	70	70
query85	1306	337	320	320
query86	354	291	290	290
query87	3229	3033	3053	3033
query88	4757	2322	2318	2318
query89	422	291	284	284
query90	1787	210	216	210
query91	167	126	128	126
query92	60	56	57	56
query93	6120	563	576	563
query94	829	212	210	210
query95	2126	1984	2053	1984
query96	655	322	317	317
query97	6495	6401	6405	6401
query98	220	197	202	197
query99	2886	997	891	891
Total cold run time: 321365 ms
Total hot run time: 213227 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.86 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef568041bb4cd58cce2e0752fb1951ce0c59bc83, data reload: false

query1	0.02	0.02	0.03
query2	0.08	0.03	0.03
query3	0.26	0.05	0.04
query4	1.78	0.07	0.08
query5	0.54	0.53	0.52
query6	1.25	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.51	0.48
query10	0.55	0.54	0.54
query11	0.14	0.09	0.09
query12	0.13	0.10	0.09
query13	0.62	0.61	0.62
query14	0.78	0.79	0.78
query15	0.77	0.76	0.76
query16	0.37	0.37	0.36
query17	1.01	1.02	1.00
query18	0.23	0.27	0.23
query19	1.88	1.82	1.85
query20	0.01	0.01	0.00
query21	15.47	0.55	0.55
query22	2.03	2.33	1.96
query23	17.28	1.00	0.85
query24	5.02	0.73	1.79
query25	0.39	0.10	0.05
query26	0.58	0.16	0.16
query27	0.04	0.03	0.04
query28	7.57	0.80	0.72
query29	12.72	2.34	2.33
query30	0.62	0.55	0.57
query31	2.81	0.39	0.37
query32	3.37	0.50	0.49
query33	3.06	3.07	3.06
query34	15.30	4.79	4.80
query35	4.91	4.90	4.86
query36	1.05	1.03	1.02
query37	0.07	0.04	0.04
query38	0.03	0.02	0.02
query39	0.03	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 103.6 s
Total hot run time: 30.86 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ef568041bb4cd58cce2e0752fb1951ce0c59bc83 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.3 seconds inserted 10000000 Rows, about 448K ops/s

@morningman morningman merged commit 22edd1c into apache:branch-2.0 Jun 18, 2025
22 of 27 checks passed
@zy-kkk zy-kkk deleted the fix_export branch July 10, 2025 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.16-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants