feat(WidgetDriver): Add to-device support (without encryption) #4963

toger5 · 2025-04-23T07:40:44Z

The widget Driver should be able to send and receive to-device events. This is useful for element call encryption keys.

This PR focusses on the widget driver and machine logic. To send/communicate the events from the widget to the driver.

It skips any encryption logic. Some of the encryption logic will be part of crypto crate and the code in the widget driver crate should be kept minimal once the crypto crate is ready.

codecov · 2025-04-30T06:25:23Z

Codecov Report

Attention: Patch coverage is 84.61538% with 18 lines in your changes missing coverage. Please review.

Project coverage is 85.87%. Comparing base (284db61) to head (aa35aad).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/matrix-sdk/src/widget/matrix.rs	76.59%	11 Missing ⚠️
crates/matrix-sdk/src/widget/machine/mod.rs	85.18%	4 Missing ⚠️
crates/matrix-sdk/src/widget/machine/driver_req.rs	71.42%	2 Missing ⚠️
crates/matrix-sdk/src/widget/filter.rs	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4963      +/-   ##
==========================================
+ Coverage   85.86%   85.87%   +0.01%     
==========================================
  Files         325      325              
  Lines       35851    35954     +103     
==========================================
+ Hits        30783    30876      +93     
- Misses       5068     5078      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A new widget filter is required to add support for to-device events. This allows to let the widget only send and receive to-device events it has negotiated capabilities for.

It consists of the following changes: - add a `NotifyNewToDeviceEvent` ToWidget request (a request that will be sent to the widget from the client when the client receives a widget action over the widget api) - add the `SendToDeviceRequest` (driver request that will be sent from the widget and asks the driver to send a ToDevice event) - add the ToDeviceActions to the required enums: `IncomingMessage`(machine), `MatrixDriverResponse`, `FromWidgetResponse`, `FromWidgetRequest`, `MatrixDriverRequestData`

stefanceriu · 2025-04-30T13:02:15Z

I'm going to let poljar handle this next week as I have very little context on it.

… events via cs api) This also hooks up the widget via the machine actions. And adds toDevice events to the subscription.

…ity. This needs to be part of the send/read capabilities so that to-device keys can be used.

poljar

I think this has a couple of flaws that need to be addressed, the biggest concern here is about the security when sending and receiving to-device messages.

I also left a bunch of less important nits, mainly to get the terminology consistent.

Please take a look at our contributing guide for our preferred commit styiling: https://github.com/matrix-org/matrix-rust-sdk/blob/main/CONTRIBUTING.md#commit-message-format.

The first commit also doesn't compile on its own, unless you want to fix that we'll have to squash merge this PR, otherwise using git bisect (if we ever need that tool) becomes hard.

Finally, I think that this deserves a changelog entry. Our contributing guide should help you with that as well: https://github.com/matrix-org/matrix-rust-sdk/blob/main/CONTRIBUTING.md#writing-changelog-entries.

poljar · 2025-05-07T12:46:07Z

crates/matrix-sdk/src/widget/filter.rs

+    }
+}
+
+impl ToDeviceEventFilter {


Why the separate impl block?

poljar · 2025-05-07T12:46:47Z

crates/matrix-sdk/src/widget/filter.rs

    State(FilterInputState<'a>),
+    // only then we can check if we can deserialize as a message like.


Suggested change

// only then we can check if we can deserialize as a message like.

// only then we can check if we can deserialize as a message-like.

poljar · 2025-05-07T12:47:00Z

crates/matrix-sdk/src/widget/filter.rs

    MessageLike(FilterInputMessageLike<'a>),
+    // ToDevice will need to be done explicitly since it looks the same as a message like.


Suggested change

// ToDevice will need to be done explicitly since it looks the same as a message like.

// ToDevice will need to be done explicitly since it looks the same as a message-like.

poljar · 2025-05-07T12:50:22Z

crates/matrix-sdk/src/widget/filter.rs

@@ -188,10 +228,35 @@ impl<'a> TryFrom<&'a Raw<AnyTimelineEvent>> for FilterInput<'a> {
    type Error = serde_json::Error;

    fn try_from(raw_event: &'a Raw<AnyTimelineEvent>) -> Result<Self, Self::Error> {
+        // FilterInput first checks if it can deserialize as a state event (state_key
+        // exists) and then as a message like event.


Suggested change

// exists) and then as a message like event.

// exists) and then as a message-like event.

poljar · 2025-05-07T13:13:59Z

crates/matrix-sdk/src/widget/filter.rs

+impl fmt::Display for ToDeviceEventFilter {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "{}", self.event_type)
+    }
+}
+


Is that Display implementation used for something? A bit of a surprising choice to just print the event type.

poljar · 2025-05-07T13:37:23Z

crates/matrix-sdk/src/widget/machine/to_widget.rs

+/// This is a "response" to the widget subscribing to the events in the room.
+#[derive(Serialize)]
+#[serde(transparent)]
+pub(crate) struct NotifyNewToDeviceEvent(pub(crate) Raw<AnyToDeviceEvent>);


This will break the compilation but I think the rename makes sense:

Suggested change

pub(crate) struct NotifyNewToDeviceEvent(pub(crate) Raw<AnyToDeviceEvent>);

pub(crate) struct NotifyNewToDeviceMessage(pub(crate) Raw<AnyToDeviceEvent>);

poljar · 2025-05-07T13:38:44Z

crates/matrix-sdk/src/widget/matrix.rs

+                        let _ = _tx.send(event_with_room_id.cast());
+                    }
+                    Err(e) => {
+                        error!("Failed to attach room id to message like event: {}", e);


Suggested change

error!("Failed to attach room id to message like event: {}", e);

error!("Failed to attach room id to message-like event: {}", e);

poljar · 2025-05-07T13:39:04Z

crates/matrix-sdk/src/widget/filter.rs

+impl<'a> TryFrom<&'a Raw<AnyToDeviceEvent>> for FilterInput<'a> {
+    type Error = serde_json::Error;
+    fn try_from(raw_event: &'a Raw<AnyToDeviceEvent>) -> Result<Self, Self::Error> {
+        // deserialize_as::<FilterInput> will first try state, message like and then


Suggested change

// deserialize_as::<FilterInput> will first try state, message like and then

// deserialize_as::<FilterInput> will first try state, message-like and then

poljar · 2025-05-07T13:55:46Z

crates/matrix-sdk/src/widget/matrix.rs

+/// Attach a room id to the event. This is needed because the widget API
+/// requires the room id to be present in the event.
+///
+/// Attach the `ecryption` flag to the event. This is needed so the widget gets


Suggested change

/// Attach the `ecryption` flag to the event. This is needed so the widget gets

/// Attach the `encryption` flag to the event. This is needed so the widget gets

Since you seem to be a typo susceptible creature, not unlike myself, let me share a small secret we have going on.

We're using typos to spell check the source tree. When we discover a new typo that somebody (mostly me) is making regularly, we head over to the non-critical typos issue on the typos repo and add it there: crate-ci/typos#1290 (comment).

This has fixed a lot of typos in the repo 😅.

poljar · 2025-05-07T14:08:27Z

crates/matrix-sdk/src/widget/matrix.rs

+/// requires the room id to be present in the event.
+///
+/// Attach the `ecryption` flag to the event. This is needed so the widget gets
+/// informed if an event is encrypted or not. Since the client is responsible


This is a particularly dangerous pattern.

Generally inserting additional metadata into the same scope that the homeserver or another user are part of is always dangerous.

One could easily imagine that the homeserver might just insert such a value themselves, or the sender themselves.

Now you'll argue that this is fine, since you're going to overwrite any value that was there. This might be true for some serializers but duplicate keys are allowed by the JSON spec, and the handling of them depends on the implementation.

It is doubly dangerous because multiple JSON implementations will be involved here, serde on the Rust side and whichever implementation the various browsers that might run the widget.

What you need to do is the following, have a struct like this:

struct ToDeviceFoo { message: Raw<AnyToDevice...>, metadata_i_add: bool, some_more_things: String }

This way the scopes we modify and the homeserver or sender control are cleanly separated.

It is fine to do this for the room_info because it's metadata we are putting back the homeserver omitted for performance reasons.

toger5 force-pushed the toger5/widget-filter-refactor branch from 9f38056 to 9504745 Compare April 23, 2025 09:18

toger5 force-pushed the toger5/widget-to-device-without-encryption branch from f6bd24a to 16171b6 Compare April 23, 2025 09:19

toger5 changed the title ~~WidgetDriver: Add to-device widget driver support~~ WidgetDriver: Add to-device widget driver support (without encryption) Apr 23, 2025

toger5 force-pushed the toger5/widget-to-device-without-encryption branch from 16171b6 to 3445127 Compare April 23, 2025 14:53

toger5 force-pushed the toger5/widget-filter-refactor branch 5 times, most recently from d1f9f49 to 50c0c2a Compare April 29, 2025 16:00

Base automatically changed from toger5/widget-filter-refactor to main April 29, 2025 16:15

toger5 mentioned this pull request Apr 30, 2025

WidgetDriver: Add io.element.call.encryption_keys to-device capability. #4990

Closed

1 task

toger5 changed the title ~~WidgetDriver: Add to-device widget driver support (without encryption)~~ WidgetDriver: Add to-device support (without encryption) Apr 30, 2025

toger5 force-pushed the toger5/widget-to-device-without-encryption branch from 3445127 to bb14445 Compare April 30, 2025 06:10

toger5 force-pushed the toger5/widget-to-device-without-encryption branch 4 times, most recently from c70a005 to 64c5426 Compare April 30, 2025 09:13

toger5 added 4 commits April 30, 2025 12:52

WidgetDriver: Introduce to-device filter.

6509fe6

A new widget filter is required to add support for to-device events. This allows to let the widget only send and receive to-device events it has negotiated capabilities for.

WidgetDriver: add toDevice capability parsing

0338b68

WidgetDriver: ffi for toDevice filter

023822e

toger5 force-pushed the toger5/widget-to-device-without-encryption branch 2 times, most recently from 5d3078c to 39ccdca Compare April 30, 2025 11:12

toger5 marked this pull request as ready for review April 30, 2025 12:44

toger5 requested a review from a team as a code owner April 30, 2025 12:44

toger5 requested review from stefanceriu and removed request for a team April 30, 2025 12:44

stefanceriu removed their request for review April 30, 2025 13:02

stefanceriu requested a review from poljar April 30, 2025 13:02

toger5 mentioned this pull request Apr 30, 2025

[TestRunner] Toger5/widget to device with encryption #4993

Draft

1 task

toger5 added 2 commits May 2, 2025 11:24

WidgetDriver: add matrix driver toDevice support (reading and sending…

79ad365

… events via cs api) This also hooks up the widget via the machine actions. And adds toDevice events to the subscription.

WidgetDriver: integration tests for the toDevice feature.

35199a7

toger5 force-pushed the toger5/widget-to-device-without-encryption branch from 9ff12fa to 35199a7 Compare May 2, 2025 09:25

WidgetDriver: Add io.element.call.encryption_keys to-device capabil…

aa35aad

…ity. This needs to be part of the send/read capabilities so that to-device keys can be used.

toger5 force-pushed the toger5/widget-to-device-without-encryption branch from ad770d9 to aa35aad Compare May 5, 2025 08:03

poljar requested changes May 7, 2025

View reviewed changes

toger5 changed the title ~~WidgetDriver: Add to-device support (without encryption)~~ feat(WidgetDriver): Add to-device support (without encryption) May 14, 2025

fkwp mentioned this pull request May 14, 2025

Support sending to_device messages in the rust sdk widget driver. (MSC3819) #4859

Open

robintown mentioned this pull request May 19, 2025

Reapply "Distinguish room state and timeline events in embedded clients" matrix-org/matrix-js-sdk#4790

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(WidgetDriver): Add to-device support (without encryption) #4963

feat(WidgetDriver): Add to-device support (without encryption) #4963

toger5 commented Apr 23, 2025 •

edited

Loading

codecov bot commented Apr 30, 2025 •

edited

Loading

stefanceriu commented Apr 30, 2025

poljar left a comment

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

poljar May 7, 2025

		State(FilterInputState<'a>),
		// only then we can check if we can deserialize as a message like.

		MessageLike(FilterInputMessageLike<'a>),
		// ToDevice will need to be done explicitly since it looks the same as a message like.

	// exists) and then as a message like event.
	// exists) and then as a message-like event.

	pub(crate) struct NotifyNewToDeviceEvent(pub(crate) Raw<AnyToDeviceEvent>);
	pub(crate) struct NotifyNewToDeviceMessage(pub(crate) Raw<AnyToDeviceEvent>);

	error!("Failed to attach room id to message like event: {}", e);
	error!("Failed to attach room id to message-like event: {}", e);

	// deserialize_as::<FilterInput> will first try state, message like and then
	// deserialize_as::<FilterInput> will first try state, message-like and then

	/// Attach the `ecryption` flag to the event. This is needed so the widget gets
	/// Attach the `encryption` flag to the event. This is needed so the widget gets

feat(WidgetDriver): Add to-device support (without encryption) #4963

Are you sure you want to change the base?

feat(WidgetDriver): Add to-device support (without encryption) #4963

Conversation

toger5 commented Apr 23, 2025 • edited Loading

codecov bot commented Apr 30, 2025 • edited Loading

Codecov Report

stefanceriu commented Apr 30, 2025

poljar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toger5 commented Apr 23, 2025 •

edited

Loading

codecov bot commented Apr 30, 2025 •

edited

Loading