[spark] Support data evolution for bucket table #6649
Conversation
shidayang
force-pushed
the
support-data-evolution-for-bucket
branch
from
6e05843 to
8c335f2
Compare
shidayang
changed the title
shidayang
force-pushed
the
support-data-evolution-for-bucket
branch
from
8c335f2 to
635b51e
Compare
| private static void validateRowTracking(TableSchema schema, CoreOptions options) { | ||
| boolean rowTrackingEnabled = options.rowTrackingEnabled(); | ||
| if (rowTrackingEnabled) { | ||
| checkArgument( |
There was a problem hiding this comment.
remove whole checkArgument.
| ``` | ||
| Notice that: | ||
| - Row tracking is only supported for unaware append tables, not for primary key tables. Which means you can't define `bucket` and `bucket-key` for the table. | ||
| - Row tracking is only supported for unaware or hash_fixed bucket append tables, not for primary key tables. |
There was a problem hiding this comment.
supported for append tables
|
And maybe you can add more tests for bucketed tables. |
shidayang
force-pushed
the
support-data-evolution-for-bucket
branch
2 times, most recently
from
02df75f to
14d57f6
Compare
@JingsongLi I've made some modifications as required and consolidated the unit tests for both bucket and non-bucket cases. PTAL |
shidayang
force-pushed
the
support-data-evolution-for-bucket
branch
from
14d57f6 to
a483524
Compare
| checkArgument( | ||
| entry.file().fileSource().isPresent(), | ||
| "This is a bug, file source field for row-tracking table must present."); | ||
| if (entry.file().fileSource().get().equals(FileSource.APPEND) |
There was a problem hiding this comment.
Because when performing compact in writer, the file will be rewritten by compact before it is committed and the rowId is generated. This scenario will result in the file generated by compact not having the rowid
There was a problem hiding this comment.
For row-tracking only tables, compact files should not be assigned new row ids, row id already in Data File.
There was a problem hiding this comment.
Maybe we should have a new bucketed append table mode here. Without considering sequence and compaction. Just like normal Append tables.
There was a problem hiding this comment.
Maybe we should have a new bucketed append table mode here. Without considering sequence and compaction. Just like normal Append tables.
good idea
3 participants
Purpose
Support data-evolution for append only fixed hash table
Tests
API and Format
Documentation