Skip to content

Lift restriction that multi-col indices only allow = constraints #2872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 688 commits into
base: mamcx/query-test-explain
Choose a base branch
from

Conversation

mamcx
Copy link
Contributor

@mamcx mamcx commented Jun 16, 2025

Description of Changes

Closes #1317

We had to disable this in #1316 but we should lift the restriction eventually for better query performance.

API and ABI breaking changes

Expected complexity level and risk

2

We optimize up to 3 columns in an index, and:

  • When all the comparisons are!= is always a full table scan
  • Multi-column indexes are only used if the query has a prefix match (ie all operators are =)
  • Else are converted to a single column index scan on the leftmost column and a filter on the rest

Testing

  • [*] Expand the testing to verify the plans to all the ops "=", ">", "<", ">=", "<=", !=
  • [*] Expand the testing to verify we return the same rows when executing the query with the above ops

rekhoff and others added 30 commits January 27, 2025 15:45
Missed updating the HandleConnect from `Subscribe("SELECT * FROM *"` to `SubscribeToAllTables()` in the updated code block on page 3.
* Updated the quickstart guide to use the new 1.0 API

* Completed quickstart rewrite

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Clarification

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Wrong type of quotes

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Update docs/sdks/typescript/quickstart.md

Co-authored-by: Phoebe Goldman <[email protected]>

* Apply suggestions from code review

Co-authored-by: Phoebe Goldman <[email protected]>
Co-authored-by: rekhoff <[email protected]>

* Address review comments

---------

Co-authored-by: Phoebe Goldman <[email protected]>
Co-authored-by: rekhoff <[email protected]>
## Description of Changes
Corresponding change to
#2177. See that PR for
more details.

~~Note that this PR only bumps the versions, but does **not** update the
DLLs. This is because SpacetimeDB will likely have further changes, so
the DLLs will just need to be updated again (in principle, they should
be updated every time we push to SpacetimeDB `master`, which isn't
really feasible. For this reason, users should not use `staging` without
having their own copy of the SpacetimeDB repo as well).~~

## API

 - [ ] This is an API breaking change to the SDK

No breaking changes.

## Requires SpacetimeDB PRs
#2177

## Testsuite
SpacetimeDB branch name: master

## Testing
- [x] CI passes with that branch name
- [x] The branch name has been changed back to `master` after that PR
merges and CI still passes

---------

Co-authored-by: Zeke Foppa <[email protected]>
## Description of Changes

Use SpacetimeDB commit hash in the cache key and skip
rebuild/reinstallation altogether if we got an exact hit.

This saves 6-8 minutes off CI time on reruns.

## API

 - [ ] This is an API breaking change to the SDK

*If the API is breaking, please state below what will break*

## Requires SpacetimeDB PRs
*List any PRs here that are required for this SDK change to work*

## Testsuite
*If you would like to run the your SDK changes in this PR against a
specific SpacetimeDB branch, specify that here. This can be a branch
name or a link to a PR.*

SpacetimeDB branch name: master

## Testing
*Write instructions for a test that you performed for this PR*

- [x] CI testing forth and back until it worked correctly.
## Description of Changes

This is the companion PR for
#2184, please see the
other PR for full description.

On the client side main changes are:

- Regenerate .NET and Unity test client bindings and test snapshot.
- Remove `IDatabaseRow` since V9 multi-tables splits data types from
actual table definitions, so those "table data types" are no longer
special. Just using `IStructuralReadWrite` in its place now.
- Add base index classes as mentioned in the other PR.
- As a sub-improvement, the non-unique index class actually does
indexing instead of iterating over the entire table like we did before.
- Remove internal-but-not-really `InternalInvokeValueDeleted` and
`InternalInvokeValueInserted` methods in favour of private events.

## API

 - [x] This is an API breaking change to the SDK

Removes some technically-visible but internal APIs.

## Requires SpacetimeDB PRs
#2184

## Testsuite
*If you would like to run the your SDK changes in this PR against a
specific SpacetimeDB branch, specify that here. This can be a branch
name or a link to a PR.*

SpacetimeDB branch name: ingvar/csharp-new-codegen

## Testing
*Write instructions for a test that you performed for this PR*

- [x] Manually tested Blackholio

---------

Co-authored-by: James Gilles <[email protected]>
* Fix C# server build process & get rid of dead link in client manifest.json

* Fix unfinished paste

* Update for new c# codegen

* Update generated code and package jsons

* Revert server-rust dependency override
## Description of Changes

Unfortunately, none of our tests currently cover this, but while working
on the V9 upgrade, I noticed that this code still relies on `type(Row)`
as a unique table identifier.

That no longer holds with multi-tables as several tables can share the
same `Row` type. In that case, subscription updates would be grouped
incorrectly and always applied to the same first table that uses `Row`
for its data storage.

This PR fixes that by using the table handle itself as a key (compared
by reference).

If transaction updates are already grouped uniquely by table, it should
be possible to simplify this code much further, but I'm not sure if such
guarantee exists, so leaving that untouched.

## API

 - [ ] This is an API breaking change to the SDK

*If the API is breaking, please state below what will break*

## Requires SpacetimeDB PRs
*List any PRs here that are required for this SDK change to work*

## Testsuite
*If you would like to run the your SDK changes in this PR against a
specific SpacetimeDB branch, specify that here. This can be a branch
name or a link to a PR.*

SpacetimeDB branch name: master

## Testing
*Write instructions for a test that you performed for this PR*

- [x] I did `dotnet test`, but as mentioned in the description, this
requires adding tests for multi-table subscriptions, which I'm afraid I
won't have time to do, so have to leave to follow-up devs.

---------

Co-authored-by: james gilles <[email protected]>
## Description of Changes

C# part of #1836
Needs to be rebased onto
clockworklabs/com.clockworklabs.spacetimedbsdk#220
once that is merged.

## API

 - [x] This is an API breaking change to the SDK

ScheduleAt is now constructed in slightly different ways.

## Requires SpacetimeDB PRs
*List any PRs here that are required for this SDK change to work*

## Testsuite
*If you would like to run the your SDK changes in this PR against a
specific SpacetimeDB branch, specify that here. This can be a branch
name or a link to a PR.*

SpacetimeDB branch name: phoebe/timestamp-special-type

## Testing
Will need an update to blackholio as well.
## Description of Changes
#2226

## API

 - [x] This is an API breaking change to the SDK

callbacks are better

## Requires SpacetimeDB PRs
#2226

## Testsuite

SpacetimeDB branch name: jgilles/csharp-better-events

## Testing

- [x] quickstart
- [x] blackholio
## Description of Changes
This seems to fix the unity testsuite, which otherwise is failing when
pointed at master.

## API

No breaking changes.

## Requires SpacetimeDB PRs
None

## Testsuite
This is meant to fix pointing at master.

SpacetimeDB branch name: master

## Testing
- [x] Unity testsuite now passes

---------

Co-authored-by: Zeke Foppa <[email protected]>
* Updated the quickstart to use the new 1.0 API

* Ran prettier

* Updated CSS

* Removed server code

* Moved out of the client dir

* Updated lockfile

* Added new workflow

* Small fixes

* Updated PR template

* Ran prettier

* Fixes to workflwo

* Workflow fixes

* Fixes the toolchain thing

* Added SpacetimeDB integration test

* Fix workflow hopefully

* Fix workflow

* Prettier

* Updated pull request template to better reflect the reality

* Fixed path in workflow

* Prettier after codegen

* Regenerate

* Say yes to deleting data

* Updated codegen to latest master of SpacetimeDB

* debug actions

* Formatting

* add README steps

* Ran lints

---------

Co-authored-by: Zeke Foppa <[email protected]>
)

* Style guide: add formatting advice for menu items

* Generalize guidance to all GUI elements, not just menu paths
* Updated the quickstart to use the new 1.0 API

* Ran prettier

* Updated CSS

* Removed server code

* Moved out of the client dir

* Updated lockfile

* Added new workflow

* Small fixes

* Updated PR template

* Ran prettier

* Fixes to workflwo

* Workflow fixes

* Fixes the toolchain thing

* Added SpacetimeDB integration test

* Fix workflow hopefully

* Fix workflow

* Prettier

* Updated pull request template to better reflect the reality

* Fixed path in workflow

* Prettier after codegen

* Regenerate

* Say yes to deleting data

* Updated codegen to latest master of SpacetimeDB

* debug actions

* Formatting

* Implemented EventContext update

* Removed unused imports

* Fix merge marker
* Add `Timestamp` to the SDK, moving it out of `client_api`

Companion to SpacetimeDB#1836

* Add `TimeDuration` class

* Add forgotten return type annotation

* 🤦

* Prettier

* Update `client_api` schema to use `TimeDuration`

* Micros, not nanos

From SpacetimeDB 072c2ea:

Prioritize BSATN and BFLATN compat with 0.12

Timestamp and TimeDuration store `micros: i64`, not `nanos: i64`.
This means that old commitlogs and snapshots should still be compatible,
assuming they don't include timestamps greater than `i64::MAX`,
as previously we used `micros: u64`. This seems unlikely.

* Ran new code gen

* Fixed rebase conflicts

---------

Co-authored-by: Tyler Cloutier <[email protected]>
* Add shared ScheduleAt type

* Lint

---------

Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
## Description of Changes

Companion to [Rename `Address` to
`ConnectionId`](#2220).
See that PR's description for more.

Like all the SDKs, this PR does not change the SDK's behavior; the SDK
still generates a connection ID locally and passes it through the HTTP
API. This is not exposed to users, and can be changed in a follow-up.

Also, use `/usr/bin/env bash` instead of `/bin/bash` in tools, for
portability reasons.

## API

 - [x] This is an API breaking change to the SDK

`Address` is renamed to `ConnectionId`.

## Requires SpacetimeDB PRs

- #2220
- 
## Testsuite
*If you would like to run the your SDK changes in this PR against a
specific SpacetimeDB branch, specify that here. This can be a branch
name or a link to a PR.*

SpacetimeDB branch name: phoebe/rename-address-to-connection-id

## Testing

- [x] Pretty much just automated testing.
- [x] @kazimuth will need to update and run Blackholio.

---------

Co-authored-by: James Gilles <[email protected]>
* Rename address to connection ID

* Format
* Now emitting UnknownTransaction if reducer name is empty string or the reducer args are not parseable

* Fixed small bug

* Lint
* Implemented the new subscriptions API

* Fixed the ../index thing

* Fixed the ../index thing for realz

* Fixes for Timestamp and TimeDuration

* Better error handling

* Fixed rebase issues

* lint

* Reduce diff

* lint
…#244)

## Description of Changes
as described

## API

no breaks

## Requires SpacetimeDB PRs
#2241

## Testsuite

SpacetimeDB branch name: jgilles/unknown_reducer

## Testing

see SpacetimeDB PR
## Description of Changes
Companion PR for the http api glowup.

## API

Not a breaking change; this catches us up to being compatible with a
breakage introduced by
- #2225.

## Requires SpacetimeDB PRs
- #2225

## Testsuite
SpacetimeDB branch name: master

## Testing
Existing CI passes (it was failing without this change since it couldn't
connect).

Co-authored-by: Zeke Foppa <[email protected]>
## Description of Changes
We were not stripping `/` from the end of URIs provided to `Connect`. We
manually append `/...` to the provided addresses, so if we don't start
by stripping trailing `/`s, we end up with `//` in the URI and we get
errors.

Addresses part of
#1551.

## API

No breaking  changes. This fixes an error case.

## Requires SpacetimeDB PRs
None

## Testsuite
SpacetimeDB branch name: master

## Testing
- [x] Tested the quickstart chat client with a host containing a
trailing `/`
```
# start a server, publish the module, send some input
# I also updated one line in `client.csproj` to use `Net8.0` because I no longer have `Net7.0` installed

$ cd examples~/quickstart/client

$ dotnet run
[I] SpacetimeDBClient: Connecting to ws://localhost:3000 quickstart-chat
C200098E is online
Connected
C2007471: hello
C2007471: godo
C2007471: asdf

$ sed -i 's/localhost:3000/localhost:3000\//' Program.cs

$ dotnet run
[I] SpacetimeDBClient: Connecting to ws://localhost:3000 quickstart-chat
C2000601 is online
Connected
C2007471: hello
C2007471: godo
C2007471: asdf

$ git diff
diff --git a/examples~/quickstart/client/Program.cs b/examples~/quickstart/client/Program.cs
index 9eb43b1..289e736 100644
--- a/examples~/quickstart/client/Program.cs
+++ b/examples~/quickstart/client/Program.cs
@@ -7,8 +7,8 @@ using System.Threading;
 using SpacetimeDB;
 using SpacetimeDB.Types;
 
-const string HOST = "http://localhost:3000";
-const string DBNAME = "chatqs";
+const string HOST = "http://localhost:3000/";
+const string DBNAME = "quickstart-chat";
 
 // our local client SpacetimeDB identity
 Identity? local_identity = null;
diff --git a/examples~/quickstart/client/client.csproj b/examples~/quickstart/client/client.csproj
index 48917cc..ab7ce44 100644
--- a/examples~/quickstart/client/client.csproj
+++ b/examples~/quickstart/client/client.csproj
@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFramework>net7.0</TargetFramework>
+    <TargetFramework>net8.0</TargetFramework>
     <CheckEolTargetFramework>false</CheckEolTargetFramework>
     <ImplicitUsings>disable</ImplicitUsings>
     <Nullable>enable</Nullable>
```

---------

Co-authored-by: Zeke Foppa <[email protected]>
## Description of Changes

Switches to Bearer authentication, which is the more proper auth schema
to use with tokens.

## API

 - [ ] This is an API breaking change to the SDK

## Requires SpacetimeDB PRs
- #2181

## Testsuite
*If you would like to run the your SDK changes in this PR against a
specific SpacetimeDB branch, specify that here. This can be a branch
name or a link to a PR.*

SpacetimeDB branch name: master

---------

Co-authored-by: Zeke Foppa <[email protected]>
Co-authored-by: rekhoff <[email protected]>
Co-authored-by: James Gilles <[email protected]>
…19)

* Fix C# server bug with overlapping circles

Update for Timestamp stuff

* Update for EventContext changes (TODO: update package .jsons once c# SDK PR is merged

* Regenerate

* Fix package .jsons

* Update for new AuthToken logic

* Update for no onUnhandledReducerError + fix package jsons

* Revert server-rust/Cargo.toml change
bfops and others added 13 commits July 16, 2025 21:21
Co-authored-by: Nathaniel Richards <[email protected]>
Co-authored-by: John Detter <[email protected]>
Co-authored-by: John Detter <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: Ingvar Stepanyan <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: Nathaniel Richards <[email protected]>
Co-authored-by: Piotr Sarnacki <[email protected]>
Co-authored-by: Phoebe Goldman <[email protected]>
Co-authored-by: Dylan Hunt <[email protected]>
Co-authored-by: Puru Vijay <[email protected]>
Co-authored-by: Kim Altintop <[email protected]>
Co-authored-by: Chip <[email protected]>
Co-authored-by: Zeke Foppa <[email protected]>
Co-authored-by: Shubham Mishra <[email protected]>
Co-authored-by: Mats Bennervall <[email protected]>
Co-authored-by: ike709 <[email protected]>
Co-authored-by: Egor Gavrilov <[email protected]>
Co-authored-by: Arrel Neumiller <[email protected]>
Co-authored-by: Muthsera <[email protected]>
Co-authored-by: james gilles <[email protected]>
Co-authored-by: rekhoff <[email protected]>
Co-authored-by: joshua-spacetime <[email protected]>
Co-authored-by: Mario Montoya <[email protected]>
Co-authored-by: Noa <[email protected]>
Co-authored-by: Mazdak Farrokhzad <[email protected]>
Co-authored-by: Steve Biedermann <[email protected]>
Co-authored-by: Oliver Davies <[email protected]>
Co-authored-by: Kane Viggers <[email protected]>
Co-authored-by: Colter Haycock <[email protected]>
Co-authored-by: AdielMag <[email protected]>
Co-authored-by: cjodo <[email protected]>
Co-authored-by: heliam1 <[email protected]>
Co-authored-by: 8Times <[email protected]>
Co-authored-by: torjusik <[email protected]>
Co-authored-by: Michael Nadeau <[email protected]>
Co-authored-by: Tamaro Skaljic <[email protected]>
Co-authored-by: Jeffrey Dallatezza <[email protected]>
Co-authored-by: Loki McKay <[email protected]>
Co-authored-by: Aaron Matthis <[email protected]>
Co-authored-by: Robin Curbelo <[email protected]>
Co-authored-by: João Saldanha Streibel <[email protected]>
Co-authored-by: Sahil Dawka <[email protected]>
Co-authored-by: Wes Sleeman <[email protected]>
Co-authored-by: Carlos Cobo <[email protected]>
Co-authored-by: John Detter <[email protected]>
Co-authored-by: John Detter <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: Zeke Foppa <[email protected]>
Co-authored-by: Ingvar Stepanyan <[email protected]>
Co-authored-by: SteveGibson <[email protected]>
Co-authored-by: Steve Gibson <[email protected]>
Co-authored-by: james gilles <[email protected]>
Co-authored-by: rekhoff <[email protected]>
Co-authored-by: WeslomPo <[email protected]>
Co-authored-by: Clockwork Labs <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: John Detter <[email protected]>
Co-authored-by: Tyler Cloutier <[email protected]>
Co-authored-by: Derek Brinkmann <[email protected]>
Co-authored-by: dbrinkmanncw <[email protected]>
Co-authored-by: John Detter <[email protected]>
Co-authored-by: SteveBoytsun <[email protected]>
Co-authored-by: Steve <[email protected]>
Co-authored-by: Alessandro Asoni <[email protected]>
Co-authored-by: Derek Brinkmann <[email protected]>
Co-authored-by: james gilles <[email protected]>
Co-authored-by: Phoebe Goldman <[email protected]>
Co-authored-by: Ingvar Stepanyan <[email protected]>
Co-authored-by: Kurtis Mullins <[email protected]>
Co-authored-by: Jeremie Pelletier <[email protected]>
Co-authored-by: Mazdak Farrokhzad <[email protected]>
Co-authored-by: Zeke Foppa <[email protected]>
Co-authored-by: Steve Boytsun <[email protected]>
Co-authored-by: Ingvar Stepanyan <[email protected]>
Co-authored-by: joshua-spacetime <[email protected]>
Co-authored-by: Noa <[email protected]>
Co-authored-by: rekhoff <[email protected]>
Co-authored-by: Daniel Kierkegaard Andersen <[email protected]>
Co-authored-by: Guribo <[email protected]>
Co-authored-by: Lisandro Crespo <[email protected]>
@mamcx mamcx force-pushed the mamcx/lift-multi-col-only-eq branch from 3e8108b to 43eb5dd Compare July 21, 2025 18:53
@mamcx mamcx force-pushed the mamcx/lift-multi-col-only-eq branch from 43eb5dd to 85679e8 Compare July 24, 2025 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Do not merge Do not merge PRs with this label without coordinating further release-any To be landed in any release window
Projects
None yet
Development

Successfully merging this pull request may close these issues.