Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SyncV2 skeleton and add migration to store serialized components #32

Merged
merged 42 commits into from
Jan 31, 2025
Merged
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
327039f
Add CBOR media type
ChrisPenner Aug 1, 2024
8487d19
Install cbor and make a basic streaming endpoint
ChrisPenner Aug 1, 2024
3754922
Stream hashes from PG
ChrisPenner Aug 2, 2024
6d69b41
Wire up test stream and get it working with conduit
ChrisPenner Aug 2, 2024
7e12608
Use Servant's Named Routes API
ChrisPenner Aug 2, 2024
854629f
Add serialized entity tables and add code to write to them.
ChrisPenner Aug 3, 2024
6837661
Include actual components in stream
ChrisPenner Aug 5, 2024
f6bb8c2
Bump unison submodule
ChrisPenner Dec 13, 2024
2ac3e3b
SyncV2 WIP
ChrisPenner Aug 12, 2024
272cf1c
WIP
ChrisPenner Aug 12, 2024
f0abad8
WIP
ChrisPenner Aug 12, 2024
a218f49
Debugging
ChrisPenner Aug 13, 2024
5862bce
Try reversing hash ordering
ChrisPenner Aug 13, 2024
19592f7
Fix compilation
ChrisPenner Dec 4, 2024
541a87c
Rename SyncStream -> SyncV2
ChrisPenner Dec 13, 2024
b8b9d55
Bump unison version
ChrisPenner Dec 16, 2024
cd55056
Update Share for new unison
ChrisPenner Dec 16, 2024
f354a4f
Debugging
ChrisPenner Dec 17, 2024
a14b424
Debugging cleanup
ChrisPenner Dec 17, 2024
7938f89
Fix up stream init settings
ChrisPenner Jan 24, 2025
6e3908b
Add background jobs for Migrating to serialized components
ChrisPenner Jan 24, 2025
bf94052
Serialise -> Serialized
ChrisPenner Jan 24, 2025
4df50e0
Fix up serialized tables
ChrisPenner Jan 24, 2025
4b33240
Remove debugging
ChrisPenner Jan 24, 2025
894ae04
Clean up dependency query
ChrisPenner Jan 24, 2025
e418855
Disable Contribution Diffs background job for now
ChrisPenner Jan 24, 2025
7c12d6a
Bump submodule
ChrisPenner Jan 24, 2025
d339350
GZip syncv2, fix gzip on syncv1
ChrisPenner Jan 24, 2025
cfce15b
Fix typo in setup query
ChrisPenner Jan 24, 2025
71f18bd
use safer transaction mode
ChrisPenner Jan 24, 2025
e2cc7d3
Ensure we copy serialized components when importing causals
ChrisPenner Jan 27, 2025
754254a
Add logging to migration
ChrisPenner Jan 27, 2025
18da43e
Split up migration into sandboxed and unsandboxed
ChrisPenner Jan 28, 2025
f6a23f0
Fixes and updates
ChrisPenner Jan 28, 2025
c1cda4d
Comment out manual steps from migration
ChrisPenner Jan 28, 2025
471d984
Remove noisy logging from migration
ChrisPenner Jan 28, 2025
9e4020d
Fix sandboxed queue population query
ChrisPenner Jan 28, 2025
7ea7a61
Add migration sanity checks
ChrisPenner Jan 29, 2025
eb9b434
Fix erroneous skipping of some components
ChrisPenner Jan 29, 2025
f81261e
Add helper queries
ChrisPenner Jan 29, 2025
d03fee0
Fix gzip for CBOR
ChrisPenner Jan 29, 2025
40e05bd
Add in user id for serialized components to allow using index
ChrisPenner Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions package.yaml
Original file line number Diff line number Diff line change
@@ -44,7 +44,9 @@ dependencies:
- bytestring
- bytes
- case-insensitive
- cborg
- clock
- conduit
- containers
- cookie
- cryptonite
@@ -94,7 +96,10 @@ dependencies:
- servant-auth
- servant-client
- servant-server
- servant-conduit
- serialise
- stm
- stm-chans
- text
- these
- time
16 changes: 16 additions & 0 deletions share-api.cabal
Original file line number Diff line number Diff line change
@@ -34,6 +34,8 @@ library
Share.BackgroundJobs.Monad
Share.BackgroundJobs.Search.DefinitionSync
Share.BackgroundJobs.Search.DefinitionSync.Types
Share.BackgroundJobs.SerializedEntitiesMigration.Queries
Share.BackgroundJobs.SerializedEntitiesMigration.Worker
Share.BackgroundJobs.Workers
Share.Branch
Share.Codebase
@@ -94,6 +96,7 @@ library
Share.Utils.Logging.Types
Share.Utils.Postgres
Share.Utils.Servant
Share.Utils.Servant.CBOR
Share.Utils.Servant.Client
Share.Utils.Servant.PathInfo
Share.Utils.Servant.RawRequest
@@ -152,6 +155,9 @@ library
Share.Web.UCM.Sync.HashJWT
Share.Web.UCM.Sync.Impl
Share.Web.UCM.Sync.Types
Share.Web.UCM.SyncV2.API
Share.Web.UCM.SyncV2.Impl
Share.Web.UCM.SyncV2.Queries
Unison.PrettyPrintEnvDecl.Postgres
Unison.Server.NameSearch.Postgres
Unison.Server.Share.Definitions
@@ -208,7 +214,9 @@ library
, bytes
, bytestring
, case-insensitive
, cborg
, clock
, conduit
, containers
, cookie
, cryptonite
@@ -252,13 +260,16 @@ library
, raven-haskell
, safe
, semialign
, serialise
, servant
, servant-auth
, servant-client
, servant-conduit
, servant-server
, share-auth
, share-utils
, stm
, stm-chans
, text
, these
, time
@@ -351,7 +362,9 @@ executable share-api
, bytes
, bytestring
, case-insensitive
, cborg
, clock
, conduit
, containers
, cookie
, cryptonite
@@ -395,14 +408,17 @@ executable share-api
, raven-haskell
, safe
, semialign
, serialise
, servant
, servant-auth
, servant-client
, servant-conduit
, servant-server
, share-api
, share-auth
, share-utils
, stm
, stm-chans
, text
, these
, time
2 changes: 2 additions & 0 deletions sql/2023-10-18-00-00_temp_entities.sql
Original file line number Diff line number Diff line change
@@ -92,6 +92,8 @@ END;
$$ LANGUAGE plpgsql;

-- Copies ALL dependencies of a causal into the codebase.
--
-- Note: This has been replaced in a later migration.
CREATE OR REPLACE FUNCTION copy_causal_into_codebase(causal_id_to_copy INTEGER, from_codebase_user_id UUID, to_codebase_user_id UUID)
RETURNS VOID AS $$
DECLARE copied_hash TEXT;
Loading