Compiler: Support table-valued functions, && operator

[simolus3]

The new sync stream compiler supports table-valued functions in FROM clauses as long as these functions only use inputs that can be derived from the current connection. This is enough to support the equivalent of table-valued parameter queries from bucket definitions. However, it is not enough to support the && (JSON array overlap) operator supported by the old sync streams implementation.

This expands support for table-valued functions, and uses that to implement &&.

Table-valued functions

Table-valued functions like json_each now support having a synced table or parameter table as inputs. To support that, the compiled sync plan is expanded as follows:

1. TableProcessors (bucket data sources and parameter index lookup creators) now have a tableValuedFunctions property describing which table-valued functions to add as inner joins when evaluating a source row. For each function, that describes the function name (e.g. json_each), input expressions derived from the source row and filters.
2. Parameter values (represented as the PartitionKey interface) are allowed to reference outputs from joined table-valued functions.
3. For parameter queries, outputs are also allowed to reference outputs from table-valued functions.

A few examples might help to demonstrate how table-valued functions work and how they're implemented:

Example 1: Static filter

Consider the stream SELECT * FROM posts WHERE 'important' IN posts.descriptions. As usual with sync streams, the IN is desugared to SELECT posts.* FROM posts, json_each(posts.descriptions) AS subquery0 WHERE subquery0.value = 'important'.

In querier_graph.ts, we resolve table-valued functions along with their input table (posts int this case). So this stream compiles into a single TableProcessor with no parameters and:

• tableValuedFunctions: [{name: 'json_each', inputs: [/* reference row.descriptions */], filters: []}]
• filters: [{/* literal('important') == tableValuedOutput(functionIndex: 0, column: 'value') */}].

Example 2: Partition on data

The stream SELECT * FROM posts WHERE subscription.parameter('tag') IN (SELECT value FROM json_each(posts.tags)) compiles to a table processor with no filters and:

• tableValuedFunctions: [{name: 'json_each', inputs: [/* reference row.tags */], filters: []}]
• parameters: [/* tableValuedOutput(functionIndex: 0, column: 'value') */]

For connections, the parameter is instantiated with subscription.parameter('tag') as usual.

Example 3: Filter on array values

This query is a bit exotic, but we support SELECT * FROM customers, json_each(customers.active_regions) AS regions WHERE region.value LIKE 'eu-%'. This translates into a table-valued function like

• tableValuedFunctions: [{name: 'json_each', inputs: [/* reference row.active_regions */], filters: [/* reference row.value LIKE literal('eu-%') */]}]

Even though the table-valued function isn't used, it still needs to be evaluated to skip customers without a matching element in the active_regions array.

Example 4: Partition on parameters

Partitions based on table-valued functions are also supported on parameter creators: SELECT users.* FROM users, orgs, json_each(orgs.members) as members WHERE users.id = members.value AND orgs.id = auth.parameter('org'). This creates a parameter lookup creator with a table-valued function and an output column referencing that function.

The && operator

Similar to how the IN operator is desugared into a joined subquery and an equals clause, we do the exact same thing for &&. The only difference is that while IN only expands the right-hand side, && expands both. && is consistently lowered into = and not considered after parsing queries.

[changeset-bot]

🦋 Changeset detected

Latest commit: f99c1be

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@powersync/service-sync-rules	Patch
@powersync/service-jpgwire	Patch
@powersync/service-core-tests	Patch
@powersync/service-core	Patch
@powersync/lib-services-framework	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-core	Patch
@powersync/service-image	Patch
test-client	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch
@powersync/service-schema	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

@simolus3 Looks like there is a merge conflict with some changes I recently made, affecting tests