Protocol changes for sync streams

[simolus3]

This is the first of two PRs to implement the new sync stream functionality in the PowerSync service. The PRs are split like this:

1. This PR refactors sync-rules so that SqlBucketDescriptor (representing parameter and data queries) isn't the only way to declare buckets. It also adds protocol-level changes required for subscription managements.
2. Another PR, WIP: New unified syntax #313, will use the new interfaces defined here to implement sync streams.

Overview

Most of this PR is related to protocol changes enabling the second feature. While most of the subscription management happens on the client, they need some help from the sync service. In particular, clients need to know why they're receiving a bucket: We want clients to be able to track progress (and also provide the usual hasSynced / lastSyncedAt fields) for individual stream subscriptions. Since progress is ultimately attached to buckets, clients need enough information to associate buckets with stream subscriptions.

Sync-rule changes

In the sync-rules package, this PR extracts most methods of SqlBucketDescriptor into the new BucketSource interface. The interface matches most methods, and adds a few new ones (e.g. to obtain a debug representation) to encapsulate functionality that used to access members in SqlBucketDescriptor directly.

When resolving queriers for a request, we add additional information (previously, we'd only pass request and token parameters):

• Whether the client is subscribing to default streams.
• For explicit stream subscriptions, an id for that subscription (generated by the client and opaque to the service) and parameters.

This information is ignored by SqlBucketDescriptor, but sync stream definitions will use it to find the right buckets and to recognize parameters.

The information returned by queriers has also been expanded: In addition to bucket names and priorities, we add:

• the name of the stream / sync rule for which the bucket has been resolved, eventually allowing clients to associate buckets with streams. While the name of the stream is technically also part of the bucket id, we want to treat ids as opaque strings instead of parsing them.
• The reason that bucket has been included, either because it's included by default (sync rules / default streams) or because there's an explicit subscription on that stream (in which case the client-side stream id is returned here).

Service changes

Most of the service changes are related to just passing the new information to sync-rules: This adds optional fields to a sync request that allow clients to specify subscribed streams.

Other changes in BucketChecksumState.ts are responsible for:

1. Letting users apply their own priorities to sync stream subscriptions.
2. Merging bucket ids: Since clients can subscribe to the same stream with different parameters, it's possible to subscribe in a way that causes streams to have overlapping results. Consider for instance a stream defined as SELECT * FROM assets WHERE id IN (request.args() -> 'asset_ids') and a client subscribing once with {'asset_id': [1]} and another time with {'asset_id': [1, 2]}. Here, both subscriptions yield the bucket assets[id=1]. We obviously don't want the bucket to be included in the checkpoint multiple times, but clients need to know that progress on bucket assets[id=1] contributes to both stream subscriptions.

[changeset-bot]

🦋 Changeset detected

Latest commit: c77b754

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-sync-rules	Minor
@powersync/service-core	Patch
@powersync/service-image	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
@powersync/service-module-core	Patch
test-client	Patch
@powersync/service-schema	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR