[meta] Threads

[bnjbvr]

The SDK is ripe for supporting threads, which allows parallel topical discussions and reduce confusion. And, it will make it possible to add support for it in embeddings, notably in ElementX, closing the gap with the Element app's feature set.

We'll update this issue as soon as we'll identify the next steps.
Here is the high-level overview of what we're planning to implement:

Interacting with threads

• feat(timeline): add functions for sending messages in threads #4819
• timeline: media attachments as (thread) replies #4835
• feat(threads): add methods to list a room's threads + list all related events #4973
  • List all the threads of a room with /threads
  • Retrieve all the thread events with /relations
• Send queue: allow sending a reply (threaded or not) to a local echo

Thread-focused timeline

A new timeline focus will be introduced that will represent a thread. Threads will include message-like events, and will be updated in real-time.

• timeline: add thread-focused timeline #4833
• feat!(timeline): Introduce an extra TimelineItemContent layer that holds aggregations #4839
• timeline: Aggregated timeline item content cleanup #4853
• refactor(timeline): bump as_variant and make more use of its pattern matching form #4854
• chore: move RedactedMessage and UnableToDecrypt to MsgLike #4860
• feat(ui): add new Thread timeline focus mode and associated events loader #5032
• How to filter out in-thread events from a timeline? Using the EventFilter might be sufficient, but if it's not we'll need a different type.
• Handle events related to an in-thread events, from sync, in real-time. Probably means finishing refactor(timeline): replace TimelineEventKind with TimelineAction #4655 so we can only observe aggregations from sync, and apply them in the thread if needs be.

Thread data model

Make it possible to save threads events in the event cache (even if they haven't been received from sync). We also want to persist and restore a thread's summary to/from persisted storage.

Some developments are happening here.

Core types

• a thread may have a ThreadSummary:
  • count (likely not the raw event count, but renderable event count)
  • latest event id (observers can use the load_or_fetch_event() method to retrieve it from the cache when needed)
  • participated: TBD
  • maybe_outdated: bool indicating whether the summary might be outdated, after a gappy sync
• for each thread (identified by room id + root event id), a ThreadEventCache, which contains a linked chunk + the thread summary.
• for each room, a RoomThreadCache, which contains a mapping of all the threads roots to their ThreadEventCache.
• the RoomEventCache contains exactly one RoomThreadCache. Separating them helps separating concerns + testing threads in isolation.

Tasks

• Reconcile or unify RoomEvents event chunks with AllEventsCache events #3886
• event cache: update storage to store the events' content in a separate table #4841
• Extract the bundled aggregation from a thread root, and feed it into the RoomEventCache, which forwards it to the ThreadEventCache ONLY if the thread was not known beforehand
  • Save the bundled thread's latest event into the event cache (don't insert into the thread linked chunk, only save it into the events table with save_event())
• Have a linked chunk per thread.
  • RoomEventCache gets a mapping of thread root id to ThreadEventCache { events: LinkedChunk<...>, } (consider using a specialized gap of ThreadGap { prev_token: Option<String>, next_token: Option<String> }).
  • Sync appends new events into the thread linked chunk in real-time.
  • Room back-pagination does NOT propagate events into the thread linked chunk. Otherwise, as soon as we have a gap in the thread, we don't know where the room-backpaginated events should go in the thread.
  • clear all the ThreadEventCaches upon gappy sync. Indeed we don't know if we missed events in that specific threads, so we don't really have a better choice at the moment (until we get an SSS extension with the list of threads that had updates)
• Allow subscribing to a thread by receiving the initial events in that thread, if known, and a stream of VectorDiff<Event> (this will power the thread-focused timeline).
• implement back-pagination at the ThreadEventCache level; this will be used by the thread-focused timeline paginate_backwards() method.

Thread persistent storage

• Generalize the SQLite backend to hold thread linked chunk. It might be sufficient that a linked chunk's key isn't a room anymore, but a generalized enum LinkedChunkKey { Room(OwnedRoomId), Thread(OwnedRoomid, OwnedEventId) }. Migration = clear all the chunks?
• Add a table in storage to store thread summaries separately from the root event; or instead of storing TimelineEvent into storage, store an augmented data structure that also includes a ThreadSummary (⚠ this second alternative would require a migration).
  • Add a method in the event cache store to upsert_thread_summary
  • Keep participated in clear, so that we can query for all the threads we've participated to.
• For each event that had more than one edit (>1 or >=1 TBD), introduce an edit linked list, to order them relative to each other, and figure out which is the actual latest.
• What to do after a gappy sync?
  • Thread summary
    • Drop them all upon gappy sync (+ notify)
    • Refresh all with /context (with 0 events before/after)
    • ❓ What about events which were not thread roots, but may be now?
  • Thread linked chunk
    • In a non-persisted world, we can clear the linked chunk of a gappy thread immediately.
    • Upon gappy sync, do a /threads query to retrieve all the threads and their latest event; if it matches what we knew from the thread linked chunk, mark it as NOT maybe_outdated. If not, start back-pagination until all the thread events are known.
      • This has been deemed impractical, because /threads can be super slow for large rooms like MatrixHQ. Instead, we could, for each known thread, run a single /relations query from the back, and keep on back-paginating until we know all the events; or let the embedder manually request to back-paginate, like we do for live main timelines.
      • Likely recompute the thread summary afterwards 👀 (Think about conflicts with sync.)

Read receipts

We want to support thread read receipts. This will not interact with unread status on rooms, yet.

• Event cache: handle read receipts and unread counts #4113
• show thread receipts
• don't attach non-thread receipts to thread events, in the live timeline
• timeline: mark a thread as read #4834