[meta] Latest Event API, the missing one

[bnjbvr]

The Latest Event represents the event that is displayed in the Room List under each room item, or displayed as a Thread Summary. It is important, and must be reactive. It follows several constraints, such as:

• Not all events are a candidate to be a Latest Event, e.g. state events aren't candidate; thus the Latest Event has to paginate through events in reverse order (from newest to oldest) to find a suitable event if any, otherwise the information that no candidate has been found must be present,
• It must represent remote and local events for a better user experience.

The Latest Event API in its current form was added a bit in an ad-hoc way, since we needed the feature but lacked the proper storage layer for events, and the proper send queue. What we can re-use though is all the logic to find a suitable event, which was not trivial to write in an efficient way knowing we have to deal with encrypted event and so on.

Today is different: We have the Event Cache API, and we have the Send Queue API. Those are the two components we need to write a proper Latest Event API! I love it when a plan comes together.

The Latest Event API in its current form is scattered in multiple crates. It's going to be hard to “migrate”, or move things together. The plan we came up with the team is to write a new API from scratch, and then migrate the components using a Latest Event one by one.

Important

Storage

A note about the storage. The Event Cache API has its own storage. The Send Queue API has its own storage. The Latest Event API will be computation only, no storage is required, and that will solve multiple problems. Right now, the Latest Event is stored in RoomInfo, which is not a good place.

Important

Thread

Another note about threads. The Latest Event API in its current form is only used by the Rooms. But we need it to work with Threads too, as it is required for Thread Summary.

Product decisions/questions

• If the latest event is an edit, show it if it edited the previous latest rendered event.
• If the latest event has been redacted, do we show "a message has been redacted", or the previous preview event?
• What counts as an interesting latest preview event?
  • We already have made some decisions there, but what about reactions?
• If there's a local echo that hasn't been sent yet (and it should've been sent, say, 8 hours ago), but there's been new messages in the room since then, what do we show: the local echo or the most up to date message?
• If there's a new in-thread message in a room, can it become the latest preview event for that room, or are only main-thread messages candidates for being the latest preview event?
  • Product proposes that only the main-timeline messages are candidates for the latest preview event for a room.

Plan

Create the Latest Event API

• Create the Latest Event API
  • The entry point is matrix_sdk::latest_events::LatestEvents
  • LatestEvents is lazy, it computes and keeps only requested LatestEvents in memory
  • LatestEvents does not run one task per room or thread; ideally we want one or two tasks to handle all our needs, so that it reduces the pressure on the system
  • Provide a LatestEvent type
  • Provide a LatestEventSubscriber type
    • It implements Stream<Item = LatestEvent>
    • ❓ It implements an “auto-drop” mechanism: when there is no more listener for all clones of the same LatestEventSubscriber, lets' drop it from LatestEvents
  • LatestEvents listens to the Event Cache for remote events
    • Provide a LatestEvents::subscribe_to_room(&self, &RoomId) -> (Option<LatestEvent>, LatestEventSubscriber) (note: Option<LatestEvent> to handle the case where there is no suitable LatestEvent)
    • Provide a LatestEvents::subscribe_to_thread(&self, &RoomId, &EventId) -> (Option<LatestEvent>, LatestEventSubscriber)
    • LatestEvents uses the Event Cache to find a suitable event
      • Be able to back-paginate in the storage
      • Be able to back-paginate in the network
  • LatestEvents listens to the Send Queue for local events
    • Send Queue has a higher priority than Event Cache, i.e. if there is a local event, it must always be used for the LatestEvent

Use the Latest Event API

• Use the Latest Event API
  • matrix_sdk_ui::room_list_service uses LatestEvents
    • room_list_service::Room::latest_event() uses LatestEvents to retrieve a single LatestEvent
      • This method no longer needs a Timeline to fetch the LatestEvent!
    • room_list_service::RoomList::entries_with_dynamic_adapter() should not need any modification (see next section)

Notify matrix_sdk::Room when the LatestEvent changes

• Notify a matrix_sdk::Room when the LatestEvent changes
  • Rename RoomInfoNotableUpdate to RoomNotableUpdate
  • Rename RoomInfoNotableUpdateReasons to RoomNotableUpdateReasons
  • LatestEvents is able to send a RoomNotableUpdate by itself, with the reason RoomNotableUpdateReasons::LATEST_EVENT

Clean up

• Clean up
  • Get rid of the code handling re-decryption of the latest preview event, in the matrix-sdk-base crate.
  • Remove the old LatestEvent API wherever it lives

See also

• Event cache: Move decryption retry in there #3872: An encrypted event is decrypted during the sync flow. Sometimes, it's not possible to decrypt an encrypted event because not all information are present. In this case, we need to re-decrypt it later. The re-decryption logic current lives in the matrix_sdk_ui::timeline, but it should move inside the Event Cache soon. How it will interact with the Latest Event API is still unclear but ultimately should be transparent (the RoomEventCache stream of updates should notify that a new event has been updated, and the machine should handle that seamlessly).

Part of #3058.

---

David (Bernini)

[mxandreas]

Regarding product/design questions. As we were tackling this in a slightly more specific context with @americanrefugee (which was to show local echos and the fact if the room has any pending or failed messages), our guiding principle was consistency between the timeline of the room and the room list - whatever is the latest event in the room, should be also shown in the room list. I think for most scenarios this works well+ from user perspective and its additional benefit is that it saves us from potentially long conversations discussing what a good alternative logic is. Or from explaining this to users if they do not follow that logic.

About the specific questions, then:

If the latest event is an edit, show it if it edited the previous latest rendered event.

Yes (I guess), although I would specify this as: if it edited the last message in the room timeline.

If the latest event has been redacted, do we show "a message has been redacted", or the previous preview event?

Message has been redacted. If this not helpful, we better offer an option to hide redacted messages at all.

What counts as an interesting latest preview event?
We already have made some decisions there, but what about reactions?

I can't any imagine any useful scenario. Showing reactions on the latest message of the timeline, but that is probably also quite far fetched.

If there's a local echo that hasn't been sent yet (and it should've been sent, say, 8 hours ago), but there's been new messages in the room since then, what do we show: the local echo or the most up to date message?

By the guideline, it is the local echo (as it should be the one that appears at the bottom of the timeline). Additionally, we would also indicate that the room has pending messages. Also, I do not think it is a very frequent scenario when you have a pending message for several hours but then you get new messages - unless you upload a gigabyte-size video, or?

If there's a new in-thread message in a room, can it become the latest preview event for that room, or are only main-thread messages candidates for being the latest preview event?

As the point of the Thread is not to "pollute" the main timeline, I think its messages should also not appear in the room list.