feat/frame scheduling #490
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When uploading memory with size X into a dmabuf, create a buffer that is square (e.g. sqrt(x) by sqrt(x)) instead of one that is X pixels wide and one pixel high. Drivers have limitations on buffer dimensions, and e.g. the intel (iris) driver ran into this limitation.
Use playbin3 for playback instead of uridecodebin. This also enables audio support. Furthermore, separate out the appsink (which uploads buffers into a flutter texture) into a separate "pseudo" gstreamer-element, which can be created using `flutter_gl_texture_sink_new`. Synchronization is reworked as well, it is now assumed the player is only accessed from a single thread, the platform thread. The callbacks called by gstreamer (which are called on internal threads) will rethread, if necessary, by posting messages to the GstBus. The buffering query is also now properly fixed. Previously it queried the playbin instead of the element that does the actual queueing (a `multiqueue`). Also, a one-shot debug logging message is added to the frame uploader, to inform the user when slower, manual dmabuf uploads are used instead of zero-copy.
Only supported since gstreamer 1.24.0 and requires some more work to implement correctly.
Also don't use conditional compilation in places that don't absolutely depend on it.
There's an issue in the gstreamer video4linux2 plugin where seeking will lead to buffer management issues, causing streaming to stop, or even memory leaks. See: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4465
- add `raw_std_method_call_from_buffer` to create a "raw_std_value" from a byte buffer & size, and check whether its encoding is valid - add `platch_respond_malformed_message_std` to send a pre-determined response if the encoding was invalid
Used by Zed to store clangd cache.
- add audio balance setter/getter - support releasing & prerolling a playback source - add separate seeking info, duration and end of stream notifiers - make gstreamer_video_player.h not depend on gstreamer headers (if possible)
Finishing unifying playback element used by audioplayers and video player plugins.
Did cause a message to be logged on every atomic commit.
On Raspberry Pi, the fix is now backported to 1.22: RPi-Distro/repo#391 This reverts commit dc0ff2a.
Was added in gstreamer 1.20, debian bullseye is still on 1.18.
Uses gstreamer 1.20.0-only API and is only used for 1.20.0 anyway.
Change notifiers should only trigger when notifier_notify is called, not on listen.
gstplayer: - allow registering callbacks for async seeks/state changes - implement gapless looping using playbin about-to-finish or segments - listen to GST_MESSAGE_EOS instead of appsink eos signal, since appsink is only used as a video sink for video playback. - query duration in GST_MESSAGE_DURATION_CHANGED message audioplayers: - actually send prepared, completed, seek completed and error events - handle incorrect file source uris by audioplayers - fix incorrect non-null terminated string printing gstreamer_video_player: - enable gapless looping by default - allow overriding it in set_looping call
player Respond with a detailed error message, that pipeline playback is not yet implemented instead of just "video player creation failed".
Disable audio for video players by default, for compatibility. Only enable when calling `createWithAudio` platform channel method.
Only diagnose flutter-pi sources by default. The diagnosis results are uploaded as an HTML workflow artifact for debugging. Also, remove the old (unused) CodeQL workflow, which had a lot of hard to disable false positives.
Mostly double-float casts, functions without arguments without an explicit `void` in the parameter list, but in platformchannel.c there were actually some memory leaks on error paths, which are now fixed. Also enable -pedantic for even more diagnostics.
drmdev: (mostly) non-stateful part. Basically a better wrapper around a drm fd. Fully mt-safe. Also properly separate kms_req from drmdev. kms_req now mostly accesses public drmdev API, instead of having access to hidden drmdev state. drm_resources: the DRM state / resources. Is stateful, but does not update itself. To keep it in sync with kernel state, one needs to listen to kernel events with drm_monitor and call drm_resources_update. drm_resources is not mt-safe and only supposed to be used on a single thread. Also add a bunch of QoL stuff to drm_resources.
To be able to specify more surface usage hints, i.e. `GBM_BO_USE_SCANOUT` and `GBM_BO_USE_RENDERING`.
Always do non-blocking atomic commits. To resolve issues with flutter trying to present frames too soon, while another frame was already sent to the kernel (i.e. comitted via KMS atomic commit), we either have to discard subsequent, too-early frames or queue them. In this case, we basically keep a 1-frame deep queue (or 2-frames, if you count the in-kernel waiting-for-scanout frame), with the 1 queued frame being displaced whenever a newer frame is available. (Similar to vulkans MAILBOX swapchain presentation mode)
Call the release callback for the commit that is no longer visible on screen before calling the scanout callback. The scanout callback might already start a new frame, and the release callback releases some resources that could be useful for building the new frame, so this helps avoid resource exhaustion.
libdrm's drmIsKMS was added in libdrm 2.4.105:
https://gitlab.freedesktop.org/mesa/libdrm/-/commit/523b3658aa8efa746417e916c987de23740ce313
Debian bullseye is still on 2.4.104, so it's unfortunately not present
there.
Fix waiting_for_scanout not being set when a frame is presented directly in the on_scanout callback, sometimes causing another frame to be presented, resulting in an EBUSY from drmModeAtomicCommit.
ardera
force-pushed
the
feat/frame-scheduling
branch
from
eea29be to
09fa863
Compare
Make frame_scheduler able to time frames, e.g. tell flutter when to start rendering a frame when it receives a frame request. For now, if no frame is currently waiting to be scanned out, this just responds to the frame request with the current frame and a 60Hz frame interval, i.e. starts the frame immediately. If a frame is waiting to be scanned out, this waits until the scanout to reply to the frame timings request. (With the timestamp of the scanout being used as the frame start time in the reply to the frame timings request) If another frame is already queued after that, the scheduler will wait until that other queued frame is scanned out to reply to the frame timings request.
ardera
force-pushed
the
feat/frame-scheduling
branch
from
09fa863 to
54cc8fd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
flutter_embedder.hflutter_embedder.h(flutter stable 3.24.0)