UI freezes with camera Feed on older iPad when using METAL renderer

[patryk-sredzinski]

Describe the bug
I was able to create simple application which just launches camera and draws it onto VideoView.

SDK Version
2.3.1

iOS/macOS Version
18.3.1 - iPad 11' 1st GEN (Intel)

Steps to Reproduce
Run the below code on older device. Works perfectly on my iPhone 15 Pro, but fails on my iPad.
(previous SDK versions worked perfectly)
(changing renderMode to sampleBuffer solves the issue too)

import UIKit
import LiveKit
import MetalKit

class ViewController: UIViewController {

    lazy var metalView: VideoView = {
        let view = VideoView()
        view.backgroundColor = .red
        view.translatesAutoresizingMaskIntoConstraints = false
        return view
    }()
    
    lazy var track = LocalVideoTrack.createCameraTrack()
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        view.addSubview(metalView)
        
        NSLayoutConstraint.activate([
            metalView.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            metalView.centerYAnchor.constraint(equalTo: view.centerYAnchor),
            metalView.widthAnchor.constraint(equalToConstant: 300),
            metalView.heightAnchor.constraint(equalToConstant: 600),
        ])
        
        Task {
            do {
                try await track.start()
                metalView.track = track
            } catch {
                print(error)
            }
            
        }
    }
}

Stack when pausing the debugger during the freeze.

[pblazej]

@patryk-sredzinski can you show the first entries (top) from the stack trace? It may be an issue with StateSync itself (Intel suggests different lock behavior, etc.).

[patryk-sredzinski]

The same code now started working for me. Also my main app did as well (same TestFlight build)... Seems like a hardware issue?
Can't reproduce it now

[pblazej]

What came to my mind is os_unfair_lock() used under the hood is not recursive, so you gotta be careful with calling state updates from state observers... However, I haven't seen any similar reports 🤞

[patryk-sredzinski]

And It broke again, I cannot see any video. My other device works well. Unfortunately the stack is empty.
When I change mode to sampleBuffer, I can see the preview

When I unpause debugger I get:
2025-03-13T17:17:42+0100 warning LiveKitSDK : [LiveKit] CameraCapturer._process(frame:capturer:device:options:) Frame processing hasn't completed yet, skipping frame...

[patryk-sredzinski]

Funny thing, when I debug METAL with:

After I am done with it and I continue execution, everything starts to work normally and iPad shows preview nicely

[hiroshihorie]

Can you try running on your iPad with release mode if it makes any difference?

[patryk-sredzinski]

I did try a TestFlight build in my app which works on iPhone and fails on iPad.
This caused me to try the SDK (as we have just migrated from older version where it was working on iPad)

So I did try to generate a minimum code to reproduce it and I did with the code above.

But to confirm - release mode do experience the same issue

[hiroshihorie]

Since you can reproduce it, can you try this branch with #530 reverted.
https://github.com/livekit/client-sdk-swift/tree/hiroshi/revert-video-processing
Thank you!

[patryk-sredzinski]

I will in a moment,
but funny thing

I wanted to screen-record the freeze
When I started screen capturing and launched this app, video was rendered properly without any issues. I tried multiple attempts to launch the app.

But the moment I tapped stop screen recording button the whole UI froze

[patryk-sredzinski]

https://github.com/livekit/client-sdk-swift/tree/hiroshi/revert-video-processing

Didn't help

[hiroshihorie]

Thank you for trying, since you mentioned "previous SDK versions worked perfectly".
Can you identify since which version this issue occurs on your "Old iPad (Intel)" ?

[patryk-sredzinski]

I just tried 2.0.19 - Still same issue, so or it is something with my iPad hardware.
Or maybe latest iOS release.

This is what I experience
https://youtu.be/vuHrTeTnZb4?si=Uq58YCbBrLQ-Rd9T

[hiroshihorie]

Thank you for the video, if it's a state lock issue, I don't think the whole iOS will become unresponsive.
Does re-starting your iPad help ?

[patryk-sredzinski]

Tried that already didn't help

[patryk-sredzinski]

That's crazy how screen recording makes the app work in background, but the moment I stop recording everything freezes and whole iPad becomes unresponsive. I hope it's just my hardware or iOS version, as I remember just a few days ago I was able to do video calls using LiveKit SDK.

I am updating to iOS 18.3.3.

another video showing in real app usage:
https://youtu.be/aGNDq3fbydg?si=O_h6rjvT_O8iIsHc - updated

[patryk-sredzinski]

And just to be clear, I haven't experienced any freezes like that on my iPad at all. The only thing that freezes is is somehow LiveKit SDK with Metal renderer. CMSampleBuffer works without issues