[skip/helpers] Add `join_one()`/`join_many()` helpers. #787

beauby · 2025-03-02T12:50:35Z

Example usage for join_one():

type User = { name: string; email: string; };
type Post = { title: string; body: string; author_id: number; };
type PostWithAuthor = { title: string; body: string; author: User; };
...
// The following turns `Post`s and `User`s into `PostWithAuthor`s.
join_one(posts, users, {
  on: "author_id",
  name: "author",
})

Example usage for join_many():

type Upvote = { post_id: number; user_id: number; };
type Post = { title: string; body: string; };
type PostWithUpvotes = {
  title: string;
  body: string;
  upvotes: { user_id: number; }[];
};
...
// The following turns `Post`s and `Upvote`s into `PostWithUpvote`s.
join_many(posts, upvotes, {
  on: "post_id",
  name: "upvotes",
})

The default type parser from the `pg` NPM library converts these to `Date` objects which then isn't handled nicely in the Skip heap. This instead just keeps the raw string, which clients can handle as they choose. Also adds test coverage of a non-lowercase column, which came up in some user code.

The issue also happens in Wasm. close SkipLabs#633

Some basic package READMEs. which will then render in the browser on npmjs.com for our packages. close SkipLabs#712

…0.0.10

This takes all of our Skip runtime packages to 0.0.10, increments all of our core `@skip-wasm/*` and `@skiplang/*` packages (some of which are already at 1.0.x, and none of which users will install, so I left out of the everything-to-0.0.10 edict), and stops versioning the test package.json.

…cer`.

Temporary disconnect hackernews lib check

- Internal packages are managed with symbolic links.

When calling `runWithGc()` without a `synchronizer`, the global lock was released after each iteration, and never re-acquired.

When calling `runWithGc()` without a `synchronizer`, the global lock was released after each iteration, and never re-acquired. The code path did not seem to be actively used, as it would systematically crash.

… tests

Increase buffer times to de-flake the expectation tests on our examples, leaving enough time for servers to spin up fully before clients start hitting them. This passed 20x in a row locally just now, but circleCI seems more inconsistent in general so testing up there as well. Marking as draft for now as I expect this may require a couple iterations.

This PR adds a git submodule for the docs_site repo and adjusts the docs workflow to update its contents, removing the need to manually sync the generated files between repos. The workflow for publishing the docs to the docs site is now (from www/README.md): ``` - $ make docs-publish ``` which will suggest to run: ``` Test locally: make docs-serve Push to live site: cd www/docs_site/; git add -A; git commit -m 'update to <commit>'; git push; cd - ``` Once the suggested command ending in `git push` is executed, the live docs site will update in a few seconds.

Before this PR the following reliably fails for me: ``` $ npm install && npm run build $ git clean -Xdf $ npm install && npm run build ``` In the final step, the build sees the symlinks created in the first step that were deleted in the second step as some sort of zombie, where `fs.exists` thinks they do not exist but `fs.symlink` fails because they do. Checking in a shell shows that `ls` is similarly confused. This PR changes the build from checking if the link exists and only if not calling symlink to creating a symlink with a temporary name and then renaming it to the desired name. The temporary names are randomly generated, and POSIX requires the rename to be atomic.

beauby · 2025-03-07T14:40:06Z

Like I said, I'm wondering whether we could make the syntax more user-friendly.

Turns out, the type constraint on EagerCollection.map, forcing the mapper's constructor parameters to be ...params: Params with Params extends DepSafe[] prevents us from using an object literal to mimic named parameters (since the object literal won't be DepSafe). I'll merge it with the initial API (constructor(other_collection, join_key, resulting_property)), as it is in helpers and not part of the core API, and we can revisit later.

Iterators are dangerous, they are used to traverse a (materialized or computed-on-the-fly) sequence. But it's Iterator-dependent whether they represent a point in the traversal or the traversal itself. In some cases, they can be cloned, reused, restarted. In others they can't. Most of the times, they can't and must be used linearly. E.g. `it.drop(1).first()` and `_ = it.drop(1); it.first()` may and may not give the same result, depending on the implementation. `next` and `sizeHint` are an exception though. This PR moves the materialization of values from `NonEmptyIterator` (Skip) to `ValuesImpl` (JS), then restricts how `NonEmptyIterator` is used.

@jberdine

…kipLabs#790) @jberdine , this should fix the excess querying you were seeing. Now that we've got framework-managed unique identifiers per-resource-instance, we can just use those as the pg_notify "channel" identifier.

The previous code relied on an assumption that `key`s were _primary_ or at least unique. But it may be useful to index on a foreign (or otherwise non-unique) key column for some data models, to skip the step of re-keying your skip collection. This PR drops that assumption and tweaks the unit test to include a table with non-unique key.

This got left out of some PR recently (looks like SkipLabs#786, my bad), just catching it up now.

Example usage for `join_one()`: ``` type User = { name: string; email: string; }; type Post = { title: string; body: string; author_id: number; }; type PostWithAuthor = { title: string; body: string; author: User; }; ... // The following turns `Post`s and `User`s into `PostWithAuthor`s. join_one(posts, users, { on: "author_id", name: "author", }) ``` Example usage for `join_many()`: ``` type Upvote = { post_id: number; user_id: number; }; type Post = { title: string; body: string; }; type PostWithUpvotes = { title: string; body: string; upvotes: { user_id: number; }[]; }; ... // The following turns `Post`s and `Upvote`s into `PostWithUpvote`s. join_many(posts, upvotes, { on: "post_id", name: "upvotes", }) ```

jberdine · 2025-03-13T16:21:35Z

skipruntime-ts/helpers/src/index.ts

@@ -14,3 +14,4 @@ export {
 export { SkipExternalService } from "./remote.js";
 export { SkipServiceBroker, fetchJSON, type Entrypoint } from "./rest.js";
 export { Count, Max, Min, Sum } from "./utils.js";
+export { join_one, join_many } from "./join.js";


Are the snake_case haters going to object?

I'm happy to rename for consistency.

jberdine · 2025-03-13T16:39:15Z

skipruntime-ts/helpers/src/join.ts

+    return values.toArray().map((v: VLeft) => {
+      const { [this.on]: key_right, ...value_left } = v;
+      const value_right = {
+        [this.name]: this.right.getUnique(key_right),


Is it obvious that this should use getUnique vs mapEntry producing an association for each value of key_right?

Not sure I understand. The getUnique() comes from the fact that this is the implementation of JoinOne. Do you mean that it is not clear whether this should be a left/right/inner join? As it stands, what we have is semantically the equivalent of having a non-nullable foreign key in left, rather than a general left join.
It could be helpful to provide a helper for the case of a nullable foreign key in left (which would make the output type Iterable<[K, Omit<VLeft, IdProperty> & Record<JoinedProperty, VRight | null>]> and replace the getUnique() with a getArray()[0].

The getUnique() comes from the fact that this is the implementation of JoinOne.

:-) I'm going in the other direction: trying to understand what JoinOne is meant to do from this code.

Do you mean that it is not clear whether this should be a left/right/inner join? As it stands, what we have is semantically the equivalent of having a non-nullable foreign key in left, rather than a general left join.

I suppose I'm thinking of this as just a multimap operation, and not in terms of sql, so my first expectation is that e.g.

h ↦ [{α, l:k}] ₗ⊔ᵣ k ↦ [v₁, v₂,..., vₙ] = h ↦ [{α, r:v₁}, {α, r:v₂},..., {α, r:vₙ}]

should hold. SQL joins have not yet become intuitive to me though, so maybe I'm just wrong.

I just don't like getUnique.

It could be helpful to provide a helper for the case of a nullable foreign key in left (which would make the output type Iterable<[K, Omit<VLeft, IdProperty> & Record<JoinedProperty, VRight | null>]> and replace the getUnique() with a getArray()[0].

Although getArray()[0] will not be equivalent if there are multiple rather than zero values.

The getUnique() comes from the fact that this is the implementation of JoinOne.

:-) I'm going in the other direction: trying to understand what JoinOne is meant to do from this code.

Do you mean that it is not clear whether this should be a left/right/inner join? As it stands, what we have is semantically the equivalent of having a non-nullable foreign key in left, rather than a general left join.

I suppose I'm thinking of this as just a multimap operation, and not in terms of sql, so my first expectation is that e.g.

h ↦ [{α, l:k}] ₗ⊔ᵣ k ↦ [v₁, v₂,..., vₙ] = h ↦ [{α, r:v₁}, {α, r:v₂},..., {α, r:vₙ}]

should hold. SQL joins have not yet become intuitive to me though, so maybe I'm just wrong.

Ok I understand now. I think there are two different notions:

what I had in mind, which basically corresponds to the mundane use case of "I have two collections of objects, the first of which (collection A) contains objects with an b_id field that corresponds to (exactly) one of the objects from the second one (collection B), and I want to build a collection of objects where the b_id field (from A) is replaced with the actual corresponding object (from B).". It implicitly assumes B is a simple map (i.e. only one value per key), and that each b_id field from A actually corresponds to an entry in B. It is basically the semantics of an SQL join with a non-null foreign key.

what you mentioned, which makes more sense from an "operations on multimaps" perspective, is a generalization of what I had in mind, where the foreign key constraint is lifted (i.e. the b_id field of A can correspond to multiple objects of B). While I think it makes sense to offer that option to users, I expect most of the time people will want the former (i.e. having more than one value per key in B would be the result of a bug, and thus the getUnique() in join_one() would rightfully warn them that something went wrong). Maybe the solution is to offer a generic way to enforce that a collection is indeed a simple map? In that case, we can use your semantic for join_one() and keep offering the right guarantees.

I just don't like getUnique.

It could be helpful to provide a helper for the case of a nullable foreign key in left (which would make the output type Iterable<[K, Omit<VLeft, IdProperty> & Record<JoinedProperty, VRight | null>]> and replace the getUnique() with a getArray()[0].

Although getArray()[0] will not be equivalent if there are multiple rather than zero values.

jberdine · 2025-03-13T17:00:13Z

skipruntime-ts/helpers/src/join.ts

+>(
+  left: EagerCollection<K, V1>,
+  right: EagerCollection<V1[IdProperty], V2>,
+  options: {


I don't understand why these 2 args are packed into a record. Just to try to have named arguments, or something else?

Yes, it's just to mimic named arguments. I'm not sure which is best, but the rationale was that two consecutive positional string arguments might be more error prone.

jberdine · 2025-03-13T17:02:56Z

skipruntime-ts/helpers/src/join.ts

+  right: EagerCollection<V1[IdProperty], V2>,
+  options: {
+    on: IdProperty;
+    name: JoinedProperty;


I'm unsure of this name name. Is there an established terminology? E.g. it was not clear to me that it had anything to do with the produced result, as opposed to selecting things from the inputs. I don't know, maybe "to"?

Agreed, name is not very helpful.

I think on/name as the Id/Joined property for both join_one and join_many is also potentially confusing...

In join_one, on is a foreign-key pointer from left into right, with the assumption that it's unique in right.

In join_many, on is a foreign-key pointer from right into left, with no uniqueness assumption.

The actual semantics seem good/useful for dealing with SQL one-to-many (join_one, where left is the "many" and right is the "one" table) and many-to-many (join_many, where left is one of the two "many" tables and right is the linking table) relationships but the naming/terminology is super tricky/under-specified here. I wonder if we could borrow that SQL relationship language here instead of talking about left/right/join (with all the connotations those carry)

i.e. maybe join_one(left,right,{on,name}) could instead be joinOneToMany({one, many, oneToManyFKey, asCol}) and join_many(left,right,{on,name}) could instead be joinManyToMany({many, linking, linkingFKey, asCol}) or something along those lines...

This line of thinking also suggests an optional third collection argument for joinMany, corresponding to the other many table -- e.g. if you wanted to have an array of user objects instead of just user IDs in the upvotes field of the test case here.

I think as is already much better than name, yes.

This line of thinking also suggests an optional third collection argument for joinMany, corresponding to the other many table -- e.g. if you wanted to have an array of user objects instead of just user IDs in the upvotes field of the test case here.

Yes, this could be an optional through option, eg.

joinManyToMany(posts, users, { through: upvotes, left_id: "post_id", // Not sure about the `left_id`/`right_id` names. right_id: "user_id", as: "upvotes", })

and keep the simple case without a "join table" as:

joinManyToMany(posts, upvotes, { // `through` and `right_id` are implicitly null/undefined here. left_id: "post_id", as: "upvotes", })

bennostein and others added 30 commits February 5, 2025 15:51

[skiprutime] Fix issue SkipLabs#633

baa0547

[skiprutime] Fix issue SkipLabs#633 (SkipLabs#729)

4fe8b1b

The issue also happens in Wasm. close SkipLabs#633

Fix Makefile typo OPT->OTP

11f2538

Add NPM package READMEs

c04fc1d

NPM Package READMEs (SkipLabs#722)

59ff552

Some basic package READMEs. which will then render in the browser on npmjs.com for our packages. close SkipLabs#712

Add hubspot tracking to docs site

80bf200

Add hubspot tracking to docs site (SkipLabs#731)

8b6438e

Bump all core package versions (@skiplang/*, @skip-wasm/*)

da2c3f9

Bump Skip runtime packages, skipping some versions to synchronize at …

32e4217

…0.0.10

Update package-lock.json

fb877f8

Bump skdb wasm test timeout

3540d0b

[examples/hackernews] Explicit Dockerfiles for db and `load_balan…

508b255

…cer`.

[examples/hackernews] Fix runService() call following SkipLabs#668.

57fe3da

[ci] Add hackernews example to CI.

e7b9319

[ci] Add hackernews example to CI. (SkipLabs#703)

14c04d5

Differentiate between internal and public packets

2cc5df0

Temporary disconnect hackernews lib check

[skdb] Fix Wasm tests

b86652f

Reduce the number of package to publish (SkipLabs#686)

c8fe93b

- Internal packages are managed with symbolic links.

[skiplang/skstore] Fix locking in runWithGcIntern.

2ea3553

When calling `runWithGc()` without a `synchronizer`, the global lock was released after each iteration, and never re-acquired.

[skiplang/skstore] Fix locking in runWithGcIntern. (SkipLabs#702)

4d7731c

When calling `runWithGc()` without a `synchronizer`, the global lock was released after each iteration, and never re-acquired. The code path did not seem to be actively used, as it would systematically crash.

Increase wait time so servers spin up fully before clients in example…

9888543

… tests

Add git submodule for api docs site

b251b8b

Config ripgrep to ignore generated docs files

af9c9e3

Install rsync, used in api docs workflow

200e918

Simplify docs workflow

12d6fbb

mbouaziz added 8 commits March 7, 2025 15:36

Use nonEmptyMap rather than map

ff39b5c

Get rid of materialized

638e209

getUnique does iterate

eeb57b5

Make sure people do not misuse NonEmptyIterator

b158678

Simplify nonEmptyReduce

faf1fc0

iter is not mutable anymore

e455139

Backward-compatibility

ddb9638

Update bootstrap

8fbb4f4

mbouaziz and others added 12 commits March 7, 2025 16:13

Add hacking section on building against local packages

d8bbbef

Add hacking section on building against local packages (SkipLabs#794)

7795f44

Use unique resource instance IDs instead of generating PG client IDs

6705780

Add PG update during unit test

237af0f

Add second table and delete during postgres unit test

036ccce

Handle non-unique keys in initial postgres payload

d0bfd09

Tweak postgres unit test to include a table with non-unique key

4a9be2c

Catch package-lock.json up to latest versions

327c346

Catch package-lock.json up to latest versions (SkipLabs#798)

82ef5bc

This got left out of some PR recently (looks like SkipLabs#786, my bad), just catching it up now.

beauby force-pushed the join-mappers branch from 309ca43 to e1e6f63 Compare March 13, 2025 07:45

beauby force-pushed the join-mappers branch from e1e6f63 to 5253862 Compare March 13, 2025 07:55

beauby changed the title ~~[skip/helpers] Add JoinOneMapper/JoinManyMapper helpers.~~ [skip/helpers] Add join_one()/join_many helpers. Mar 13, 2025

beauby changed the title ~~[skip/helpers] Add join_one()/join_many helpers.~~ [skip/helpers] Add join_one()/join_many() helpers. Mar 13, 2025

bennostein approved these changes Mar 13, 2025

View reviewed changes

jberdine reviewed Mar 13, 2025

View reviewed changes

wip

bb33918

mbouaziz force-pushed the main branch from 234d593 to c37efc5 Compare March 27, 2025 11:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[skip/helpers] Add `join_one()`/`join_many()` helpers. #787

[skip/helpers] Add `join_one()`/`join_many()` helpers. #787

Uh oh!

beauby commented Mar 2, 2025 •

edited

Loading

Uh oh!

beauby commented Mar 7, 2025

Uh oh!

jberdine Mar 13, 2025

Uh oh!

beauby Mar 14, 2025

Uh oh!

jberdine Mar 13, 2025

Uh oh!

beauby Mar 14, 2025

Uh oh!

jberdine Mar 14, 2025

Uh oh!

beauby Mar 14, 2025

Uh oh!

jberdine Mar 13, 2025

Uh oh!

beauby Mar 14, 2025

Uh oh!

jberdine Mar 13, 2025

Uh oh!

beauby Mar 14, 2025

Uh oh!

bennostein Mar 14, 2025 •

edited

Loading

Uh oh!

bennostein Mar 14, 2025

Uh oh!

beauby Mar 14, 2025

Uh oh!

Uh oh!

[skip/helpers] Add join_one()/join_many() helpers. #787

Are you sure you want to change the base?

[skip/helpers] Add join_one()/join_many() helpers. #787

Uh oh!

Conversation

beauby commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beauby commented Mar 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bennostein Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

[skip/helpers] Add `join_one()`/`join_many()` helpers. #787

[skip/helpers] Add `join_one()`/`join_many()` helpers. #787

beauby commented Mar 2, 2025 •

edited

Loading

bennostein Mar 14, 2025 •

edited

Loading