Not working with Scenes

[gavrichards]

I've had to convert my AppDelegate to Swift, because my radio station app uses react-native-carplay, which now requires use of iOS Scenes which in turns requires a Swift AppDelegate.

Everything is working fine except for react-native-siri-shortcut (it was working fine before this).

I can add a shortcut still, but when I say "Hey Siri, play [name of station]" - the app opens, but getInitialShortcut returns null. Same goes for using addShortcutListener while the app is open - the event is never triggered.

Here's what AppDelegate.swift looks like. I used Swiftify to convert react-native-siri-shortcut's bit (and others) to Swift.

import UIKit
import CarPlay
import React

...

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate, RCTBridgeDelegate, UNUserNotificationCenterDelegate {

  var window: UIWindow?
  var bridge: RCTBridge?;
  var rootView: RCTRootView?;

  static var shared: AppDelegate { return UIApplication.shared.delegate as! AppDelegate }

  func sourceURL(for bridge: RCTBridge!) -> URL! {
    #if DEBUG
    return RCTBundleURLProvider.sharedSettings().jsBundleURL(forBundleRoot: "index");
    #else
    return Bundle.main.url(forResource:"main", withExtension:"jsbundle")
    #endif
  }

  func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
    
    ...
    
    self.bridge = RCTBridge(delegate: self, launchOptions: launchOptions)

    let rootView = RCTRootView(bridge: self.bridge!, moduleName: "aiirmobile", initialProperties: nil)
    
    if #available(iOS 13.0, *) {
        rootView.backgroundColor = UIColor.systemBackground
    } else {
        rootView.backgroundColor = UIColor.white
    }
    
    self.rootView = rootView

    ...

    return true
  }

  func application(_ application: UIApplication, configurationForConnecting connectingSceneSession: UISceneSession, options: UIScene.ConnectionOptions) -> UISceneConfiguration {
    if (connectingSceneSession.role == UISceneSession.Role.carTemplateApplication) {
      let scene =  UISceneConfiguration(name: "CarPlay", sessionRole: connectingSceneSession.role)
      scene.delegateClass = CarSceneDelegate.self
      return scene
    } else {
      let scene =  UISceneConfiguration(name: "Phone", sessionRole: connectingSceneSession.role)
      scene.delegateClass = PhoneSceneDelegate.self
      return scene
    }
  }

  func application(_ application: UIApplication, didDiscardSceneSessions sceneSessions: Set<UISceneSession>) {
  }

  ...

  // Added for react-native-siri-shortcut
  func application(
      _ application: UIApplication,
      continue userActivity: NSUserActivity,
      restorationHandler: @escaping ([UIUserActivityRestoring]?) -> Void
  ) -> Bool {
      return RNSSSiriShortcuts.application(application, continue: userActivity, restorationHandler: restorationHandler)
  }
}

Then in myapp-Bridging-Header.h I have:

#import <React/RCTBridge.h>
#import <React/RCTRootView.h>
#import <React/RCTBundleURLProvider.h>
#import <React/RCTAppSetupUtils.h>

#import <RNSiriShortcuts/RNSiriShortcuts.h>
#import "RNCarPlay.h"

Any thoughts or guidance would be greatly appreciated!

[Gustash]

Is there any data inside func application( _ application: UIApplication, continue userActivity: NSUserActivity, restorationHandler: @escaping ([UIUserActivityRestoring]?) -> Void ) at all?

[gavrichards]

So when I do getInitialShortcut I just get null. Is that what you mean?

[Gustash]

Can you replicate on a clean project and link it here? This is a tricky one to debug

[gavrichards]

Yes. I have just tried to set up the example project, but when I do npm install I receive this error:

example % npm install
chruby: unknown Ruby: 2.7.4
npm ERR! code EUNSUPPORTEDPROTOCOL
npm ERR! Unsupported URL Type "link:": link:..

I can see in package.json it has "react-native-siri-shortcut": "link:.." so I presume this is the reason.
Any ideas what to do here?

[Gustash]

It should work out of the box, seems like a possible issue with your dev environment

[Gustash]

Try yarn instead of npm

[gavrichards]

Ok yarn worked, thanks.

[gavrichards]

Ok I'm trying to run the example in the simulator now. I've done yarn install and pod install, but I'm getting this:

[gavrichards]

Update - I got this running, by deleting and cloning from scratch, and this time not doing yarn install in the root of the project, only in the example directory.

I then converted the AppDelegate to Swift.

And what do you know, it just works still.

Flip back to my app, and it doesn't. So I'm baffled, and now on a mission to compare the two.

[Gustash]

Seems like a specific issue with your app, unfortunately. I'll leave this open for now, but if/when you find a solution, it would be very appreciated if you could add a comment with it in case other people have the same issue you're having in the future :)

[gavrichards]

Thanks for leaving it open, I will definitely give an update once I land on the reason.

[gavrichards]

Ok I have landed on the cause through a process of elimination, but I don't yet understand why it's happening.

It stops working at the point you introduce "scenes" to AppDelegate.

In order to support CarPlay, you have to have separate scenes for Phone versus CarPlay.

If you delete the CarPlay scene and move the code from the Phone scene back into the main AppDelegate, Siri Shortcuts start working again - getInitialShortcut returns an object of info rather than null.

Is that enough information to investigate with? Are there any known incompatibiltiy issues with Scenes for this package?

[Gustash]

I don't know, that's a very specific use case that's arguably out of the scope of this package. I would rather not blow up the scope of this package in order to support another third party package.

If you do figure out the root cause, and a solution, then a PR detailing the why and how would be appreciated. I don't currently have the time to modify the library for that use case, so that would be something you'd need to figure out on your own and possibly send upstream here for other people.

[gavrichards]

I haven't explained it very well, but it's actually not specific to the CarPlay package. It happens when you simply make use of Scenes, and you only need one scene for it to happen. I will share a fork which demonstrates this asap.

[gavrichards]

Ok I've got this together. Two branches:

• One that converts the AppDelegate to Swift, but doesn't use Scenes - Siri Shortcuts work fine here
• The other does that but also uses Scenes - If you add a Siri Shortcut in this one, close the app and then re-open it using the shortcut, you'll notice you don't get the payload

I'm not sure if there's a better way of sharing these or demonstrating the differences?

[Gustash]

Thank you for the example project. Unfortunately, as I've said, I barely have time to maintain this repo with bug fixes / support for new RN releases, so I don't have the time to debug issues for a very specific use-case.

If you want to debug this and try to reach a conclusion/solution on what the implementation change needs to be to support this use-case, the best place to start looking is here. This is the method that handles a getInitialShortcut request from JS

Edit: I will leave this open until a solution is found or your research deems this as impossible to implement. Hopefully the former

[gavrichards]

Thanks. I think I have found the issue, it's described here: https://nemecek.be/blog/20/ios-13-launchoptions-always-nil-this-is-the-reason-solution
So just a case of figuring out how to use this new way of getting the launch options now. I'll keep digging into this.

[gavrichards]

This is also a helpful article for explaining the separation of AppDelegate and SceneDelegate:
https://adapptor.com.au/blog/enhance-existing-apps-with-carplay

So far I'm struggling to:

• figure out how to pass the scene's connection options into the bridge, so they can be picked up by this library when getting the initial shortcut. This needs to happen once when the app is opened, regardless of which scene is being used.
• understand why listening for shortcuts while the app is open has also stopped working, which presumably is nothing to do with launch/connection options, but more to do with the chunk of code this library asks you to add to AppDelegate.

[Gustash]

I have my doubts that a solution can be achieved here. The API for Scenes seems completely different from the AppDelegate API, and not even React Native is fully geared up to support it (as you've seen with the limitation on storing that information in the JS bridge)

[gavrichards]

Well good news. I have got this working!

I would like to share the solution with you, but I'm wondering what the best form for a PR would be.

As the AppDelegate is now in Swift, it wouldn't make sense to modify the existing example - I think there needs to be a separate Swift/Scenes example. It can be a duplicate of the existing example project, just with the necessary changes.

The problem with this approach is you won't be easily able to see the differences in a PR. But I'm not sure of an alternative?

In terms of changes to the actual Siri Shortcut library itself, these are purely additions to ios/RNSSSiriShortcuts.h and ios/RNSSSiriShortcuts.m - so it should be possible to merge these without any problems caused to people who don't need to use Scenes.

[Gustash]

Well good news. I have got this working!

I would like to share the solution with you, but I'm wondering what the best form for a PR would be.

As the AppDelegate is now in Swift, it wouldn't make sense to modify the existing example - I think there needs to be a separate Swift/Scenes example. It can be a duplicate of the existing example project, just with the necessary changes.

The problem with this approach is you won't be easily able to see the differences in a PR. But I'm not sure of an alternative?

In terms of changes to the actual Siri Shortcut library itself, these are purely additions to ios/RNSSSiriShortcuts.h and ios/RNSSSiriShortcuts.m - so it should be possible to merge these without any problems caused to people who don't need to use Scenes.

Amazing, thank you for your hard work. A new example project for scenes is perfectly fine, just note on the PR which files in the example project are relevant to these changes.

Whenever you can, submit a PR with these fixes please 😄

[gavrichards]

Here we are: #105
You can see the affected files in the example project, here: e415429

[Gustash]

@gavrichards this has been released on v3.2.0. Thanks for all your hard work!

As a heads up, I've also gone ahead an extracted the connection options -> launch options bridge initialization logic to it's own convenienve RCTBridge initializer, so you should probably use that instead.

You can use it like this:

let bridge = RCTBridge(delegate: appDelegate, connectionOptions: connectionOptions)

[gavrichards]

Oh that's interesting, thanks. I would have had no idea how to do that, or whether it's a good idea. Thanks!

[Gustash]

Oh that's interesting, thanks. I would have had no idea how to do that, or whether it's a good idea. Thanks!

Tbf, manually creating launch options from connection options is probably also not the safest idea in general. But in React Native land, we don't really go by the rules lol

[gavrichards]

No, I hated having to do that. But just couldn't see another way. The two things are not compatible, but until React Native officially issue a way to pass connection options to the bridge (which I figure once Scenes becomes more widely adopted, they'll have to address), I think it's the only way.

[Gustash]

No, I hated having to do that. But just couldn't see another way. The two things are not compatible, but until React Native officially issue a way to pass connection options to the bridge (which I figure once Scenes becomes more widely adopted, they'll have to address), I think it's the only way.

You say that but they still don't even support multiple windows afaik, and iPad support is non-existent pretty much.

I do hope they start supporting new APIs, but so far it feels like they're mostly fine with the current implementation and will probably keep it until it's deprecated (if it ever is)

[gavrichards]

There is one bug that I couldn't fix with this work, which is probably worth sharing, although I feel like it's an iOS / CarPlay issue, not this package or React Native.

Prerequisites:

• You use the Swift AppDelegate and Scenes
• You have already added a Siri shortcut in your app on your phone
• Connect phone to CarPlay, or use CarPlay simulator
• App is not already running on phone or CarPlay

Actions:

• You say the Siri command to open the app
• The app opens on CarPlay
• connectionOptions does not contain userActivities as expected, so you can't do anything specific on app startup
• The CarPlay Scene delegate's "continue" method is called immediately. This is only expected when the app is already open. It's too early to handle it as the app hasn't properly started yet.

I think this is an iOS bug, because if you don't connect to CarPlay and just try the Siri command on your phone, connectionOptions has the userActivities and you can handle things on app startup as expected.

[Gustash]

Yeah, that seems like a bug on the OS or API side tbh. Btw, I forgot to add the new example project's folder to .npmignore, so the size of the library in versions 3.2.0..3.2.1 is too big.

You should install version 3.2.2 which fixes this issue, sorry about that

[gavrichards]

Hi @Gustash
I'm back on this thread because I have a feeling the changes here with the RCTBridge initializer have broken something with push notifications.
You may recall we had to make a method for converting launch options to connection options.
My app uses push notifications, and I need to be able to get the payload from the notification when the app is launched from tapping a notification. Since these changes, nothing now comes through to the app.

If I add a log to the scene delegate's willConnectTo handler:

class PhoneSceneDelegate: UIResponder, UIWindowSceneDelegate {
  var window: UIWindow?
  func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
    guard let appDelegate = (UIApplication.shared.delegate as? AppDelegate) else { return }
    guard let windowScene = (scene as? UIWindowScene) else { return }
    
    let logger = Logger();
    let s = String(describing: connectionOptions)
    logger.log("DEBUG \(s, privacy: .public)");
...

Then close the app, send a push notification, tap on it, this is what is logged:

DEBUG {
    "_UISceneConnectionOptionsNotificationResponsesKey" = "{(\n    <UNNotificationResponse: 0x280845ad0; actionIdentifier: com.apple.UNNotificationDefaultActionIdentifier, notification: <UNNotification: 0x280845b00; source: com.myapp date: 2022-12-01 15:21:07 +0000, request: <UNNotificationRequest: 0x280849470; identifier: C78D7536-8B1B-49DB-901A-EDCF9FAE5C18, content: <UNNotificationContent: 0x12a005260; title: <redacted>, subtitle: (null), body: <redacted>, summaryArgument: (null), summaryArgumentCount: 0, categoryIdentifier: , launchImageName: , threadIdentifier: , attachments: (\n), badge: (null), sound: (null), realert: 0, interruptionLevel: 1, relevanceScore: 0.00, filterCriteria: (null), trigger: <UNPushNotificationTrigger: 0x280448290; contentAvailable: NO, mutableContent: NO>>, intents: (\n)>>\n)}";
}

But when I try to get the notification within the app, I just get undefined. This was not the case before these changes.

So I think the RCTBridge convenience method needs modifying somehow, to ensure this information still comes through. Would you be able to assist?
Thank you!

EDIT: I've added some logs into the RCTBridge modifier and can see it is returning here, when there's push notification info, so this explains ths issue:

[gavrichards]

@Gustash fixed: #114
Any chance this could be merged and released please? :)