[camera_windows] Restore image streaming support #9772
Conversation
…tform (flutter#7951)" This reverts commit b3a5e33.
The logic is synchronous, so there is no need to have the `@async` annotation here. Addresses: flutter#7067 (comment)
When image stream frames are being received from Media Foundation,
it appears to use a separate thread when calling the `OnSample`
callback, which results in this error from:
The 'plugins.flutter.io/camera_windows/imageStream' channel sent a
message from native to Flutter on a non-platform thread. Platform
channel messages must be sent on the platform thread. Failure to do
so may result in data loss or crashes, and must be fixed in the
plugin or application code creating that channel.
In order to ensure that we are only ever using the `EventChannel` from
a platform thread, create a `TaskRunner` that is backed by a hidden
window to submit the frames to the image stream.
Based on the suggestion in this comment:
flutter/flutter#134346 (comment)
Instead of a global `EventSink` that is re-used for each stream, create a dedicated `EventChannel` for each capture stream. The channel gets an unique name like `plugins.flutter.io/camera_android/imageStream/<cameraId>` for each camera instance. Addresses: flutter#7067 (comment)
We don’t really need to keep track of the subscription and controller in instance variables.
Given that subscribing to the `EventChannel` tells the capture controller to start the image stream and cancelling the subscription tells it to stop, we don’t actually need a separate method in `CameraPlatform` for stopping the stream.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request restores image streaming support for the Windows camera plugin. The implementation introduces a task runner to safely post frame data from the media foundation thread to the main thread, and uses an event channel to stream the data to the Dart side. The changes are well-structured and include corresponding tests. However, there is a critical issue in how the frame data is prepared and sent on the native side, which will prevent the code from compiling and cause a mismatch with what the Dart side expects. My review includes a fix for this issue.
| if (image_stream_sink_) { | ||
| // Create a strongly-typed frame data object using Pigeon | ||
| flutter::EncodableList data_list; | ||
| std::vector<uint8_t> buffer_data(buffer, buffer + data_length); | ||
| for (uint8_t byte : buffer_data) { | ||
| data_list.push_back(flutter::EncodableValue(static_cast<int64_t>(byte))); | ||
| } | ||
|
|
||
| PlatformFrameData frame_data(data_list, | ||
| static_cast<int64_t>(preview_frame_width_), | ||
| static_cast<int64_t>(preview_frame_height_), | ||
| static_cast<int64_t>(data_length)); | ||
|
|
||
| // Use CameraApi::GetCodec() to properly encode the Pigeon-defined class | ||
| // This ensures the frame data is serialized using the same codec as other | ||
| // CameraApi messages | ||
| flutter::EncodableValue encoded_frame = | ||
| CameraApi::GetCodec().EncodeMessage(frame_data.ToEncodableList()); | ||
|
|
||
| task_runner_->EnqueueTask([weak_sink = std::weak_ptr(image_stream_sink_), | ||
| encoded_frame = std::move(encoded_frame)]() { | ||
| std::shared_ptr<flutter::EventSink<flutter::EncodableValue>> sink = | ||
| weak_sink.lock(); | ||
| if (sink) { | ||
| sink->Success(encoded_frame); | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
The current implementation for sending frame data over the event channel is incorrect and will not compile. The EncodeMessage method returns a byte buffer, not an EncodableValue. Additionally, the Dart side expects a Map of frame data, but the native code attempts to send a PlatformFrameData object encoded for Pigeon, which is not compatible with the StandardMethodCodec used for the event channel.
To fix this, you should construct a flutter::EncodableMap with the frame data, which is what the Dart side's cameraImageFromPlatformData function expects. This aligns with the StandardMethodCodec used for the EventChannel.
Also, the current way of creating the byte list is inefficient. You can directly create an EncodableValue from a std::vector<uint8_t>.
if (image_stream_sink_) {
flutter::EncodableMap frame_data;
frame_data[flutter::EncodableValue("data")] =
flutter::EncodableValue(std::vector<uint8_t>(buffer, buffer + data_length));
frame_data[flutter::EncodableValue("width")] =
flutter::EncodableValue(static_cast<int64_t>(preview_frame_width_));
frame_data[flutter::EncodableValue("height")] =
flutter::EncodableValue(static_cast<int64_t>(preview_frame_height_));
task_runner_->EnqueueTask(
[weak_sink = std::weak_ptr(image_stream_sink_),
frame = flutter::EncodableValue(frame_data)]() {
std::shared_ptr<flutter::EventSink<flutter::EncodableValue>> sink =
weak_sink.lock();
if (sink) {
sink->Success(frame);
}
});
}
|
Tests failed, reverted to draft. Will resolve shortly. |
|
Thanks for the contribution, but going forward, please don't check boxes in the checklist that you haven't actually done. The purpose of the checklist is to guide contributors through required steps, not to simply check the boxes. |
Restores support for streaming images from the camera(s) in Windows.
I built this directly off the work done by @liff (thank you!) in !8234
It hopefully resolves the review feedback from there.
From the work done before in !8234, I have updated:
Fixes flutter/flutter#97542.
Pre-Review Checklist
[shared_preferences]pubspec.yamlwith an appropriate new version according to the pub versioning philosophy, or I have commented below to indicate which version change exemption this PR falls under1.CHANGELOG.mdto add a description of the change, following repository CHANGELOG style, or I have commented below to indicate which CHANGELOG exemption this PR falls under1.///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.Footnotes
Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. ↩ ↩2 ↩3