Skip to content

Conversation

@fcfangcc
Copy link
Contributor

@fcfangcc fcfangcc commented Dec 15, 2025

Purpose

This PR adds support for Iceberg partition transforms in the Iceberg sink. Previously, only identity partitioning was supported. With this change, users can utilize standard Iceberg transforms like year, month, day, hour, bucket, and truncate when defining partition keys.

Changes

  • IcebergDataSinkFactory: Updated to parse partition keys containing transforms from the sink.partition-key configuration.
  • IcebergMetadataApplier: Implemented logic to apply partition transforms when creating new Iceberg tables.
  • IcebergDataSinkOptions: Updated documentation to include examples of partition transforms.
  • Tests: Added IcebergDataSinkFactoryTest cases to verify that partition specs are correctly generated for various transforms.

Usage

Users can now specify partition keys with transforms in the sink.partition-key option.
Example:

sink.partition-key = "testdb.table1:year(create_time);testdb.table2:bucket[16](id)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants