feat: manage ClickHouse named collections from connector lifecycle

[royendo]

Backend half of the CH named-collections feature. The frontend half (CH source templates rewritten to reference s3(rill_<connector>, url='...') and similar) lives on royendo/template-connectors-v2-frontend (#9329) — that branch's templates emit SQL that depends on this PR's named collections existing on the CH server. Backend should land first.

What this does

When a Rill connector resource of an applicable driver type (s3, gcs, azure, mysql, postgres) reconciles in a project that uses ClickHouse as the OLAP engine, this code creates a CH named collection called rill_<connector_name> populated from the connector's resolved config. On connector delete it drops the named collection. DuckDB-OLAP projects are unaffected.

This is analogous to how the DuckDB driver creates TEMPORARY SECRETs from connectors, but the lifecycle differs: CH named collections are cluster-wide and persistent, so creation is bound to the connector resource rather than per model execution.

Files

• runtime/drivers/clickhouse/named_collections.go (370 lines) — driver field-mapping registry, CREATE/DROP NAMED COLLECTION SQL builders, permission probe, model-SQL reference detection.
• runtime/drivers/clickhouse/named_collections_test.go — unit tests for the registry, builders, and reference detection.
• runtime/drivers/clickhouse/model_executor_self.go — auto-detection hook in selfToSelfExecutor.Execute; warns when a model references rill_<conn> but no matching named collection exists.
• runtime/reconcilers/connector.go — syncClickHouseNamedCollection wired into the connector create + delete paths.
• runtime/testruntime/testruntime.go — new NewInstanceWithClickhouseFiles helper for tests that mutate project files at runtime.
• runtime/connector_named_collection_test.go — end-to-end test covering create → update → model query → delete, gated by testmode.Expensive and using the existing ch_cluster_2S_2R/ cluster fixture.

Naming convention

The named collection identifier is rill_<connector_name> — fixed by cross-PR convention with the frontend templates on #9329.

Things to look at first

1. Reconcile hook — syncClickHouseNamedCollection in runtime/reconcilers/connector.go acquires the OLAP handle on every connector reconcile. Worth confirming this is the right architectural seam vs hooking closer to the existing UpdateInstanceConnector plumbing.
2. Permission probe — CheckNamedCollectionAdmin does a CREATE + DROP on a probe collection rather than reading system.users, since granted permissions in CH are role-dependent and not portably queryable. Adds ~2 round-trips per connector reconcile; could memoize per *Connection if it becomes hot-path noise.
3. Field-name compatibility with frontend templates — the registry uses canonical CH field names (access_key_id, secret_access_key, host, database, user, etc.). Worth a sanity check against feat: rewire add-data modal to use template RPCs (template-connectors v2, PR 3) #9329's template SQL (s3(rill_foo, url='...'), postgresql(rill_foo, table='...'), etc.) before merge.

Open items flagged by implementation

• GCS native creds: only HMAC is mappable to a CH named collection. Right now the reconciler hard-fails if only google_application_credentials is set (no HMAC). Could be a warning + skip instead — open to either.
• url table-function detection: the regex includes url(...), but https-typed connectors aren't currently in the supported-driver registry, so a url(rill_https, ...) reference would auto-detect-warn without a corresponding NC. Either remove url from detection or extend the registry to handle https.
• Probe collection cleanup on cluster — if create succeeds on one shard and drop fails (network blip), a rill_probe_* collection is left behind. Currently surfaced as an error. Periodic GC would be nice but out of scope here.

Test instructions

go test -count=1 -run TestClickHouseNamedCollectionLifecycle ./runtime/

Test is gated by testmode.Expensive. Uses real CH via testcontainers. Covers connector create → NC appears with expected fields, edit → NC updated, model with s3(rill_…, …) reference resolves cleanly, connector delete → NC dropped. Cluster path (ON CLUSTER) exercised via the cluster fixture.

Checklist:

• Covered by tests
• Ran it and it works as intended
• Reviewed the diff before requesting a review
• Checked for unhandled edge cases
• Linked the issues it closes
• Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
• Intend to cherry-pick into the release branch
• I'm proud of this work!

---

Developed in collaboration with Claude Code