Skip to content

feat(event cache store): add an IndexedDB implementation #4617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
6955428
Modify the matrix-sdk and add feature flag for the event cache store
ospfranco Jan 30, 2025
b8bfd3e
Create the basic struct to initialize an event_cache_store
ospfranco Jan 30, 2025
c2761d5
Almost working instantiation of event cache store
ospfranco Jan 30, 2025
3c6ca6a
Fix deps on feature indexeddb
ospfranco Jan 31, 2025
7eb8946
Add license text at top
ospfranco Jan 31, 2025
64e68ba
Copy try_take_leased_lock from crypto_store impl
ospfranco Jan 31, 2025
1328003
Add missing methods from EventCacheTrait
ospfranco Jan 31, 2025
eebe00e
Fix a bunch of errors, move error to it's own file
ospfranco Feb 3, 2025
e49bda6
Fix final errors around creating the event cache store
ospfranco Feb 3, 2025
1b1029d
First insertion into indexeddb
ospfranco Feb 3, 2025
c317bd8
Merge branch 'main' into oscar/indexeddb-event-cache-store
ospfranco Feb 4, 2025
0f8ebf1
Finish implementing insert_chunk
ospfranco Feb 4, 2025
ee40ddd
Comment unused code, finish implementing insert_chunk
ospfranco Feb 4, 2025
73713ac
Get rid of name param
ospfranco Feb 4, 2025
d3b6f87
Insert GAP chunk
ospfranco Feb 4, 2025
8528522
Tried to implement insert gap, but missing encode_value on serializer
ospfranco Feb 4, 2025
7622be4
Implement insert gap
ospfranco Feb 6, 2025
01cb5ba
remove chunk deletes from LINKED_CHUNKS store, missing related entities
ospfranco Feb 6, 2025
8f2b427
Fix some compilation errors
ospfranco Feb 10, 2025
722bac2
PushItems
ospfranco Feb 10, 2025
4ca221f
PushItems
ospfranco Feb 10, 2025
0e05ac8
RemoteItem
ospfranco Feb 10, 2025
28647b6
Half implementation of DetachLastItems
ospfranco Feb 10, 2025
eee0bad
Remove deletion
ospfranco Feb 10, 2025
0b0e15e
Add placeholder for deletion
ospfranco Feb 10, 2025
993283c
Change previous implementations to use serializer
ospfranco Feb 11, 2025
d755cc0
Change previous implementations to use serializer
ospfranco Feb 11, 2025
abee3ab
Do not interate through get_all_with_key
ospfranco Feb 11, 2025
06b0395
Fix encoded room id as ref
ospfranco Feb 13, 2025
ae25b35
finish DetachLastItems
ospfranco Feb 13, 2025
4dcf22a
PushItems now uses TimeEventForCache struct
ospfranco Feb 13, 2025
610860b
Implement Update::Clear
ospfranco Feb 13, 2025
ebda277
Fixes on initialization of event-cache store
ospfranco Feb 17, 2025
1af016f
WIP, fixes for creating serializable objects
ospfranco Feb 17, 2025
6d8b405
More fixes on serialization and deserialization
ospfranco Feb 17, 2025
4179750
Working retrieving of events
ospfranco Feb 20, 2025
6c7b729
Update key handling
ospfranco Feb 21, 2025
c686f90
Fix ranges
ospfranco Feb 21, 2025
8cfe19c
Add stub for tests
ospfranco Feb 24, 2025
18c6287
Add indexeddbeventcachestoreerror to OpenStoreError
ospfranco Feb 25, 2025
b6a8e08
Implement clear_all_rooms_chunks
ospfranco Feb 25, 2025
533b1c3
Working media storage
ospfranco Feb 25, 2025
4b7f73d
Working media storage
ospfranco Feb 25, 2025
9e34ac0
Fixed clear command
ospfranco Feb 25, 2025
a28f40b
Enable more tests
ospfranco Feb 26, 2025
a8082ba
Turn on media_integration_tests
ospfranco Feb 26, 2025
8906c94
Partial implementation of EventCacheMediaStore
ospfranco Feb 26, 2025
824c183
EventCacheMediaStore compiles but fails tests
ospfranco Feb 27, 2025
372ac98
EventCacheMediaStore compiles but fails tests
ospfranco Feb 27, 2025
d83b6f2
Base tests passing again
ospfranco Feb 27, 2025
191cf48
Base tests passing again
ospfranco Feb 27, 2025
7b1ba9e
implement clean_up_media_cache_inner
ospfranco Feb 27, 2025
762dba4
implement clean_up_media_cache_inner
ospfranco Feb 27, 2025
fc576ee
Pass more tests for EventCacheMediaStore
ospfranco Feb 27, 2025
93ff358
working tests
ospfranco Feb 28, 2025
ad6284b
working tests
ospfranco Feb 28, 2025
8e6cc59
Fix linked chunk update when next chunk is not there
ospfranco Feb 28, 2025
db030c7
Fix linked chunk update when next chunk is not there
ospfranco Feb 28, 2025
866e2fa
Minor fixes
ospfranco Feb 28, 2025
9219a68
Minor fixes
ospfranco Feb 28, 2025
22308f5
Fix handle linked chunk updates in a single transaction
ospfranco Feb 28, 2025
4912477
Fix handle linked chunk updates in a single transaction
ospfranco Feb 28, 2025
69b45c7
All tests passing
ospfranco Feb 28, 2025
b219c9b
properly clear all data on clear_all_rooms
ospfranco Feb 28, 2025
7cd8fd4
Rename Chunk to ChunkForCache
ospfranco Mar 1, 2025
99048c2
clean up code
ospfranco Mar 1, 2025
46ae727
Rework keys to make encryption work
ospfranco Mar 3, 2025
ab61ffe
Do not encrypt event indexes to allow for sorted queries
ospfranco Mar 3, 2025
81930a1
Keep order of chunk identifier
ospfranco Mar 3, 2025
ab4afbc
Clean up code
ospfranco Mar 3, 2025
d390e07
Add encrypted tests
ospfranco Mar 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ url = "2.5.4"
uuid = "1.12.1"
vodozemac = { version = "0.9.0", features = ["insecure-pk-encryption"] }
wasm-bindgen = "0.2.84"
wasm-bindgen-futures = "0.4.24"
wasm-bindgen-test = "0.3.33"
web-sys = "0.3.69"
wiremock = "0.6.2"
Expand Down
8 changes: 6 additions & 2 deletions crates/matrix-sdk-indexeddb/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,9 @@ default-target = "wasm32-unknown-unknown"
rustdoc-args = ["--cfg", "docsrs"]

[features]
default = ["e2e-encryption", "state-store"]
default = ["e2e-encryption", "state-store", "event-cache-store"]
state-store = ["dep:matrix-sdk-base", "growable-bloom-filter"]
event-cache-store = ["dep:matrix-sdk-base", "dep:matrix-sdk-crypto"]
e2e-encryption = ["dep:matrix-sdk-crypto"]
testing = ["matrix-sdk-crypto?/testing"]

Expand All @@ -32,13 +33,16 @@ matrix-sdk-crypto = { workspace = true, features = ["js"], optional = true }
matrix-sdk-store-encryption = { workspace = true }
ruma = { workspace = true }
serde = { workspace = true }
rmp-serde = { workspace = true }
serde_json = { workspace = true }
serde-wasm-bindgen = "0.6.5"
thiserror = { workspace = true }
tokio = { workspace = true }
tracing = { workspace = true }
wasm-bindgen = { workspace = true }
web-sys = { workspace = true, features = ["IdbKeyRange"] }
wasm-bindgen-futures = { workspace = true }
gloo-timers = { version = "0.3", features = ["futures"] }
web-sys = { workspace = true, features = ["IdbKeyRange", "Blob", "BlobPropertyBag"] }
hkdf = { workspace = true }
zeroize = { workspace = true }
sha2 = { workspace = true }
Expand Down
72 changes: 72 additions & 0 deletions crates/matrix-sdk-indexeddb/src/event_cache_store/builder.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
use super::{
migrations::MigrationConflictStrategy, operations::get_media_retention_policy,
IndexeddbEventCacheStore, Result,
};
use crate::event_cache_store::{
indexeddb_serializer::IndexeddbSerializer, migrations::open_and_upgrade_db,
};
use matrix_sdk_base::event_cache::store::media::{MediaRetentionPolicy, MediaService};
use matrix_sdk_store_encryption::StoreCipher;
use std::sync::Arc;

/// Builder for [`IndexeddbEventCacheStore`]
// #[derive(Debug)] // TODO StoreCipher cannot be derived
pub struct IndexeddbEventCacheStoreBuilder {
name: Option<String>,
store_cipher: Option<Arc<StoreCipher>>,
migration_conflict_strategy: MigrationConflictStrategy,
}

impl IndexeddbEventCacheStoreBuilder {
pub fn new() -> Self {
Self {
name: None,
store_cipher: None,
migration_conflict_strategy: MigrationConflictStrategy::BackupAndDrop,
}
}

pub fn name(mut self, name: String) -> Self {
self.name = Some(name);
self
}

pub fn store_cipher(mut self, store_cipher: Arc<StoreCipher>) -> Self {
self.store_cipher = Some(store_cipher);
self
}

/// The strategy to use when a merge conflict is found.
///
/// See [`MigrationConflictStrategy`] for details.
pub fn migration_conflict_strategy(mut self, value: MigrationConflictStrategy) -> Self {
self.migration_conflict_strategy = value;
self
}

pub async fn build(self) -> Result<IndexeddbEventCacheStore> {
// let migration_strategy = self.migration_conflict_strategy.clone();
let name = self.name.unwrap_or_else(|| "event_cache".to_owned());

let serializer = IndexeddbSerializer::new(self.store_cipher.clone());
let inner = open_and_upgrade_db(&name, &serializer).await?;

let media_service = MediaService::new();
let policy: Option<MediaRetentionPolicy> = get_media_retention_policy(&inner).await?;
media_service.restore(policy);

let store = IndexeddbEventCacheStore {
inner,
store_cipher: self.store_cipher,
serializer,
media_service: Arc::new(media_service),
};
Ok(store)
}
}

impl Default for IndexeddbEventCacheStoreBuilder {
fn default() -> Self {
Self::new()
}
}
83 changes: 83 additions & 0 deletions crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
use matrix_sdk_base::{event_cache::store::EventCacheStoreError, StoreError};
use matrix_sdk_crypto::store::CryptoStoreError;
use matrix_sdk_store_encryption::Error as EncryptionError;

#[derive(Debug, thiserror::Error)]
pub enum IndexeddbEventCacheStoreError {
#[error(transparent)]
Json(#[from] serde_json::Error),
#[error(transparent)]
Encryption(#[from] EncryptionError),
#[error("DomException {name} ({code}): {message}")]
DomException { name: String, message: String, code: u16 },
#[error(transparent)]
StoreError(#[from] StoreError),
#[error("Can't migrate {name} from {old_version} to {new_version} without deleting data. See MigrationConflictStrategy for ways to configure.")]
MigrationConflict { name: String, old_version: u32, new_version: u32 },
#[error(transparent)]
CryptoStoreError(#[from] CryptoStoreError),
#[error("Uknown Chunk Type {0}")]

Check warning on line 19 in crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

View workflow job for this annotation

GitHub Actions / Spell Check with Typos

"Uknown" should be "Unknown".
UnknownChunkType(String),
}

impl From<web_sys::DomException> for IndexeddbEventCacheStoreError {
fn from(frm: web_sys::DomException) -> IndexeddbEventCacheStoreError {
IndexeddbEventCacheStoreError::DomException {
name: frm.name(),
message: frm.message(),
code: frm.code(),
}
}
}

impl From<IndexeddbEventCacheStoreError> for StoreError {
fn from(e: IndexeddbEventCacheStoreError) -> Self {
match e {
IndexeddbEventCacheStoreError::Json(e) => StoreError::Json(e),
IndexeddbEventCacheStoreError::StoreError(e) => e,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this compile, I'm surprised it doesn't need a wrapper for StoreError here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using the command Jonas gave and it compiles with that, it will not pass the tests but the compiler shows no errors left:

cargo xtask ci wasm

IndexeddbEventCacheStoreError::Encryption(e) => StoreError::Encryption(e),
_ => StoreError::backend(e),
}
}
}

impl From<IndexeddbEventCacheStoreError> for EventCacheStoreError {
fn from(e: IndexeddbEventCacheStoreError) -> Self {
match e {
IndexeddbEventCacheStoreError::Json(e) => EventCacheStoreError::Serialization(e),
IndexeddbEventCacheStoreError::StoreError(e) => {
EventCacheStoreError::Backend(Box::new(e))
}
IndexeddbEventCacheStoreError::Encryption(e) => EventCacheStoreError::Encryption(e),
_ => EventCacheStoreError::backend(e),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backend => Backend, right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}
}
}

impl From<IndexeddbEventCacheStoreError> for CryptoStoreError {
fn from(frm: IndexeddbEventCacheStoreError) -> CryptoStoreError {
match frm {
IndexeddbEventCacheStoreError::Json(e) => CryptoStoreError::Serialization(e),
IndexeddbEventCacheStoreError::CryptoStoreError(e) => e,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think this needs a wrapper as well, right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know, copied what was there from the state_store, this all were required there I think

_ => CryptoStoreError::backend(frm),
}
}
}

impl From<serde_wasm_bindgen::Error> for IndexeddbEventCacheStoreError {
fn from(e: serde_wasm_bindgen::Error) -> Self {
IndexeddbEventCacheStoreError::Json(serde::de::Error::custom(e.to_string()))
}
}

impl From<rmp_serde::encode::Error> for IndexeddbEventCacheStoreError {
fn from(e: rmp_serde::encode::Error) -> Self {
IndexeddbEventCacheStoreError::Json(serde::ser::Error::custom(e.to_string()))
}
}

impl From<rmp_serde::decode::Error> for IndexeddbEventCacheStoreError {
fn from(e: rmp_serde::decode::Error) -> Self {
IndexeddbEventCacheStoreError::Json(serde::de::Error::custom(e.to_string()))
}
}
Loading
Loading