[Question] Possible extensions on table topic feature for ducklake

[dumbNickname]

Heyo,

I hope I do not spam your issues, so please close this one as soon as you think the question simply does not match your product strategy. However I was simply too curious about the response to not try to ask it.

Would there be any interest to support a standard such as ducklake (https://ducklake.select/) next to iceberg (or maybe over some common abstraction) for the table topic feature?

Best Regards,
Bart

[Gezi-lzq]

Hi, thanks for your great question! We appreciate your curiosity and thoughtful suggestion.

The core design philosophy of the Table Topic feature is to leverage open table formats, making data from the streaming engine more accessible for analysis. Therefore, we are certainly open to integrating other formats like DuckLake or Delta Lake in the future.

However, whether now is the right time to integrate other open table formats is something we need to consider carefully. We believe the decision should be driven by practical use cases and the maturity of these formats. We will keep a close eye on the ecosystem and community demand to determine the best time for such integrations.

Thanks again for bringing this up

[dumbNickname]

Thanks a lot for such a kind and quick response!

We started looking at both data formats in our team and it is hard to disagree that ducklake seems to have many things well thought through and some benefits over iceberg (while also being an open standard). Unfortunately pretty early to say if they will be one of exotic things or maybe the next "big" thing for data pipelines. We are also pretty early in the research and still with many discussions ahead, but the design kind of feels right. Especially nicely fits what you did in automq with write ahead log - they explore the option (I say explore as it is still marked as experimental in their docs and only for use with duckdb - while other parts are said to be free to run on postgre) to use the database to batch things before writes to parquet files: https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining (maybe can be used to reduce file concatenation and parquet optimizations needed if you need frequent commits but later also efficiency when accessing the data)

Short background of my question (as it was not marked as "off topic :P): Ducklake seems to reduce the operational overhead while keeping architecture very similar to iceberg. According to my understanding they replace all metadata files with a data base (which the catalog needs anyway), what opens some new possibilities for them, reduces the amount of operations on metadata files and cascading relationships on reading those plus more operations can be transactional etc.

I totally understand you need to plan wise your efforts and the overall direction for the product. Then lets see where this ends up 😄

[Gezi-lzq]

Thanks for sharing the additional context; it's very insightful.👍

I personally agree and also admire DuckLake's approach. Handing metadata management back to the database, rather than building a complex, file-based metadata system on object storage, is a great design choice. The data_inlining feature is also very well-suited for high-frequency, small-batch data updates.

However, from a project perspective, we still need to carefully observe and plan how we would invest our resources. We really appreciate your suggestions and will keep an eye on how things develop. Let's see where it goes. 🚀

[SUBHANGANI22]

Thanks for the detailed discussion , this thread is really helpful.

From a contributor/user perspective, even if there are no near-term plans to support DuckLake, it might be useful to clarify the current abstraction boundaries of the Table Topic implementation. For example, which parts of the pipeline are tightly coupled to Iceberg (metadata handling, commit logic, schema evolution, etc.) and which parts are intentionally format-agnostic.

Having that documented could make it easier for the community to experiment with alternative table formats on their own, without committing the core project to additional maintenance or roadmap guarantees.

Really appreciate the transparency around the design trade-offs here.