[feat](seq mapping) uniq table supports multi-stream merging through sequence mapping #54936
+1,662
−37
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HonestManXin
requested review from
CalvinKirs,
dataroaring,
gavinchou,
morningman and
yiguolei
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
HonestManXin
force-pushed
the
seq_mapping
branch
2 times, most recently
from
049b532 to
cfe00a4
Compare
Contributor
Author
|
run buildall |
HonestManXin
force-pushed
the
seq_mapping
branch
from
cfe00a4 to
a264265
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 34750 ms |
TPC-DS: Total hot run time: 185399 ms |
ClickBench: Total hot run time: 32.47 s |
HonestManXin
force-pushed
the
seq_mapping
branch
from
a264265 to
637b19e
Compare
HonestManXin
force-pushed
the
seq_mapping
branch
from
637b19e to
9b45113
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 34821 ms |
TPC-DS: Total hot run time: 189967 ms |
ClickBench: Total hot run time: 29.65 s |
HonestManXin
force-pushed
the
seq_mapping
branch
2 times, most recently
from
5df34d4 to
1882828
Compare
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run feut |
Contributor
Author
|
run beut |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run cloud_p0 |
Contributor
Author
|
run feut |
Contributor
Author
|
run nonConcurrent |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run feut |
Contributor
Author
|
run nonConcurrent |
Contributor
Author
|
run cloud_p0 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
1 similar comment
Contributor
FE Regression Coverage ReportIncrement line coverage |
HonestManXin
force-pushed
the
seq_mapping
branch
from
7046f2c to
4338cd7
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-DS: Total hot run time: 190450 ms |
ClickBench: Total hot run time: 30.7 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
dataroaring
reviewed
Oct 17, 2025
| qt_sql "select * from $tableName;" | ||
| qt_sql "select a, b, c, d, e from $tableName;" | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Please add cases for other table types, like duplicate and aggregate.
dataroaring
reviewed
Oct 17, 2025
| columns.forEach(x -> columnName.add(x.getName().toLowerCase())); | ||
| resultSeqMap = new ArrayList<>(); | ||
| for (Map.Entry<String, List<String>> entry : columnSeqMapping.entrySet()) { | ||
| int sequenceColumnIdx = columnName.indexOf(entry.getKey().toLowerCase()); |
Contributor
There was a problem hiding this comment.
we'd better use column unique id here.
dataroaring
reviewed
Oct 17, 2025
| private Type sequenceType; | ||
| @Getter | ||
| @Setter | ||
| private Map<String, List<String>> columnSeqMapping = null; |
Contributor
There was a problem hiding this comment.
Please add comment for key and value, and we'd better use column UniqueId for column.
dataroaring
reviewed
Oct 17, 2025
| }, insertSql, true,"${tbName}") | ||
|
|
||
| sql """ DROP TABLE IF EXISTS ${tbName} """ | ||
| } |
Contributor
There was a problem hiding this comment.
add a case in which there is no seqmap, then adding a seqmap to the table.
dataroaring
reviewed
Oct 17, 2025
| using PatternToIndex = std::unordered_map<std::string, std::vector<TabletIndexPtr>>; | ||
| std::unordered_map<int32_t, PatternToIndex> _index_by_unique_id_with_pattern; | ||
| std::unordered_map<uint32_t /* seq cid */, std::vector<uint32_t> /* value cids */> _seq_map; | ||
| std::unordered_map<uint32_t /* cid */, uint32_t /* sequence */> _value_to_seq; |
Contributor
There was a problem hiding this comment.
Need a better name and comment.
HonestManXin
force-pushed
the
seq_mapping
branch
from
4338cd7 to
4aaf0b2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In some business scenarios, business operations need to update different columns in the same wide table through two or more data streams. For example, one data stream will write in real time to update some fields of this table; another data stream will perform imports on demand to update other columns of this table. During the update process, both data stream jobs need to ensure the order of replacement; and during queries, data from all columns should be queryable.
documentation: apache/doris-website#2954