ERROR [Error: UndefinedErrorMultilingualConfiguration]

[YashYash]

Description

Thanks for creating this package! Can't wait to get this working and productionize it. Running into the issue below.

Testing an expo dev client app.
Tested on an ios simulator . On device I got a different error. I created a separate issue for that.

Package versions

"react-native-executorch": "^0.4.7"
"react-native": "0.79.5"
"expo": "~53.0.13"
"expo-dev-client": "~5.2.2"
"@speechmatics/expo-two-way-audio": "^0.1.2" // using this library since it supports echo cancellation

I have been spending some time trying to debug this error. My current flow is:

1. Turn on the microphone to start recording
2. Call streamingTranscribe(STREAMING_ACTION.START)
3. on each mic recording chunk, I call onChunk which calls: streamingTranscribe(STREAMING_ACTION.DATA,
   Array.from(data),
   SpeechToTextLanguage.English,
   )

The streamTranscribeError error below is:

[Error: UndefinedErrorModuleNotLoaded]

This prints our repetitively when trying to transcribe each chunk.

The other error I am getting whenever the app hot reloads is:

[Error: UndefinedErrorUndefinedErrorThe package 'react-native-executorch' doesn't seem to be linked. Make sure:

- You have run 'pod install'
- You rebuilt the app after installing the package
- You are not using Expo Go
]

Here is the hook I am working on creating:

import { useCallback } from 'react'
import { Platform, PermissionsAndroid } from 'react-native'
import {
  SpeechToTextLanguage,
  STREAMING_ACTION,
  useSpeechToText,
} from 'react-native-executorch'
import {
  initialize as initializeTwoWayAudio,
  setMicrophoneModeIOS,
  toggleRecording,
  useExpoTwoWayAudioEventListener,
  useIsRecording,
} from '@speechmatics/expo-two-way-audio'

export type SpeechToTextOptions = {
  onSegment?: (text: string) => void
  onError?: (err: Error) => void
  onReady?: () => void
  language?: string
  streamingConfig?: "fast" | "balanced" | "quality" | undefined
}

type MicrophoneDataEvent = { data: Uint8Array }

export function useSpeechTranscribe({
  onSegment,
  onError,
  language = 'en',
  streamingConfig = 'balanced',
}: SpeechToTextOptions = {}) {
  const {
    streamingTranscribe,
    isGenerating,
    configureStreaming,
    error: streamTranscribeError,
    sequence,
    isReady,
    isGenerating: isLoading,
    downloadProgress,
  } = useSpeechToText({
    modelName: 'whisper',
    windowSize: 3,
    overlapSeconds: 1.2,
    streamingConfig,
  })
  const isListening = useIsRecording()
  if (streamTranscribeError) {
    console.log('#### STREAMING TRANSCRIBE ERROR', streamTranscribeError)
  }

  // Request microphone permission
  const requestMicrophonePermission = useCallback(async () => {
    if (Platform.OS === 'android') {
      const granted = await PermissionsAndroid.request(
        PermissionsAndroid.PERMISSIONS.RECORD_AUDIO,
        {
          title: 'Microphone Permission',
          message: '*** needs access to your microphone to enable *** voice mode',
          buttonNeutral: 'Ask Me Later',
          buttonNegative: 'Cancel',
          buttonPositive: 'OK',
        }
      )
      if (granted !== PermissionsAndroid.RESULTS.GRANTED) {
        throw new Error('Microphone permission not granted')
      }
    }
  }, [])

  const onChunk = (data: number[]) => {
    streamingTranscribe(
      STREAMING_ACTION.DATA,
      data,
      SpeechToTextLanguage.English,
    )
  }

  useExpoTwoWayAudioEventListener('onMicrophoneData', async (
    event: MicrophoneDataEvent,
  ) => {
    if (event && event.data) {
      const uint8 = event.data
      const int16 = new Int16Array(uint8.buffer, uint8.byteOffset, uint8.byteLength / 2)
      const floatArray = Array.from(int16, v => v / 32768)
      onChunk(floatArray)
    }
  })

  const startListening = useCallback(async () => {
    if (!isReady) return
    try {
      await requestMicrophonePermission()
      await initializeTwoWayAudio()
      streamingTranscribe(STREAMING_ACTION.START)
      toggleRecording(true)
    } catch (err: any) {
      onError?.(err instanceof Error ? err : new Error(String(err)))
    }
  }, [requestMicrophonePermission, onError, language, onSegment])

  const sitMicMode = useCallback(() => {
    setMicrophoneModeIOS()
  }, [])

  const stopListening = useCallback(() => {
    toggleRecording(false)
    streamingTranscribe(
      STREAMING_ACTION.STOP,
      undefined,
      SpeechToTextLanguage.English,
    )
  }, [])

  // Nothing logging here.
  console.log(sequence)

  return {
    startListening,
    stopListening,
    sitMicMode,
    configureStreaming,
    isReady,
    isGenerating,
    isLoading,
    downloadProgress,
    sequence,
    streamTranscribeError,
    isListening,
  }
}

Thanks!

Steps to reproduce

1. Turn on the microphone to start recording
2. Call streamingTranscribe(STREAMING_ACTION.START)
3. on each mic recording chunk, I call onChunk which calls: streamingTranscribe(STREAMING_ACTION.DATA,
   Array.from(data),
   SpeechToTextLanguage.English,
   )

Snack or a link to a repository

No response

React Native Executorch version

^0.4.7

React Native version

0.79.5

Platforms

iOS

JavaScript runtime

Hermes

Workflow

Expo Dev Client

Architecture

Fabric (New Architecture)

Build type

Debug mode

Device

Real device

Device model

iPhone 15 Pro Max

AI model

whisper

Performance logs

No response

Acknowledgements

Yes

[msluszniak]

I think that this feedback is very valuable because it not only shows that there is problem in STT but also in error messages, e.g. [Error: UndefinedErrorUndefinedErrorThe package 'react-native-executorch' ... is completely uninformative. Probably someone will take that in a few days ;)

[mkopcins]

Let me look into this issue. I can see one error in your code, you are using english only whisper model but you are passing audioLanguage argument to streamingTranscribe. Either change model to whisperMultilingual or don't pass the audioLanguage. whisper model is english only and thus there is no need to pass the language :)

[msluszniak]

I guess in this line,

react-native-executorch/packages/react-native-executorch/src/Error.ts

Line 61 in 99e0d7f

return ETErrorMessage + message;

it should be changed to: return ETErrorMessage + " " + message; or return ETErrorMessage + ": " + message; But I'm not sure from where the concatenation of UndefinedErrorUndefinedError comes.

@mkopcins any idea?

[mkopcins]

isn't that from the main branch? you should look back to tag 0.4.7. We made a couple of changes to error handling on main so this is not current

[msluszniak]

If I checked correctly, this mistaken concatenation is not fixed neither on main nor in 0.4.7 tag

Main linked above, here is 0.4.7:
https://github.com/software-mansion/react-native-executorch/blob/v0.4.7/src/Error.ts#L59

[mkopcins]

yes, you are right, concatenation needs to be fixed, also we shouldn't return UndefinedError in the message as it is simply useless

[mkopcins]

@YashYash The error you are seeing happens when you try to call streamingTranscribe before model is loaded and ready (isReady is false). Another problem I am seeing in your code is that in this callback const startListening = useCallback(async () => { you are not passing isReady in the dependency array. I would also add a safety check for isReady in onChunk function

[YashYash]

Thanks for having a look! I was actually reading through the code and realized that I should not be passing SpeechToTextLanguage.English if I am using 'whisper' or 'moonshine'. The original error I got is no longer happening when I am building with eas. Which is great. I am occasionally getting this error: [Error: UndefinedErrorModuleNotLoaded], even when isReady is true. Which is strange because I think this error is thrown only when you attempt to call streamTranscribe when isReady is false. But now transcribing is working!

I did notice that it is being pretty slow. I think it is taking close to. 12 seconds for it to start returning a sequence. Do you recommend I use a smaller model? Do you have one that you recommend?

[YashYash]

@mkopcins since I posted this issue, I did add the isReady dependency. I still get the UndefinedErrorModuleNotLoaded error. Here is a code snippet:

  const startListening = useCallback(async () => {
    if (!isReady) return
    try {
      await requestMicrophonePermission()
      await initializeTwoWayAudio()
      streamingTranscribe(STREAMING_ACTION.START)
      toggleRecording(true)
    } catch (err: any) {
      onError?.(err instanceof Error ? err : new Error(String(err)))
    }
  }, [requestMicrophonePermission, onError, language, onSegment, isReady])

Here even though isReady is true, when it gets to streamingTranscribe(STREAMING_ACTION.START) and calls it, the UndefinedErrorModuleNotLoaded error is returned. But transcribing still works. So maybe that error is firing unnecessarily. But thanks a lot for having a look.

[YashYash]

I did notice that it is being pretty slow. I think it is taking close to. 12 seconds...

I see that with balanced the window size is 12 seconds. So that explains the 12 seconds.

[mkopcins]

I did notice that it is being pretty slow. I think it is taking close to. 12 seconds for it to start returning a sequence. Do you recommend I use a smaller model? Do you have one that you recommend?

Are you running this on device or on simulator? This is not what I expect for iPhone 15 Pro Max. Generally speaking whisper is much slower to moonshine for shorter audio sequences. This is because for whisper we HAVE TO pad the audio with silence to 30 seconds, so the main time difference between various streamingConfig and windowSize is how long you need to wait for the audio from the microphone (if you set windowSize to 10 seconds you naturally need to wait at least 10 seconds for the audio before you can even start transcribing it). Considering you are using english only right now and the windowSize is so short I would personally recommend starting with moonshine model (this model can process audio chunks of various duration so should be significantly faster than whisper for this config)

[mkopcins]

I see that with balanced the window size is 12 seconds. So that explains the 12 seconds.

This should not be the case, since you are overriding it with explicitly passing in the windowSize value (actually you should be seeing warning that the config is being overwritten windowSize and overlapSeconds overrides values from streamingConfig balanced.)

[mkopcins]

Here even though isReady is true, when it gets to streamingTranscribe(STREAMING_ACTION.START) and calls it, the UndefinedErrorModuleNotLoaded error is returned. But transcribing still works. So maybe that error is firing unnecessarily. But thanks a lot for having a look.

Sounds very weird, have you added check for isReady in onChunk function? isReady should never change back to false unless you change the model. Also error is only thrown in one place in the code and that is in this line
if (!this.isReady) throw Error(getError(ETError.ModuleNotLoaded));
I will debug this a bit more later, maybe onChunk is being called in weird places? I am not familiar with @speechmatics/expo-two-way-audio library

[YashYash]

Thank you for explaining 🙏 . I will use moonshine then. I am experimenting with different window sizes now to see if I can pick the right window size and overlap seconds. Even with a window size of 3, it takes some time (7 to 10s) for it to start returning a sequence. I am wondering now if it is the speechmatics/expo-two-way-audio library. I will give. react-native-live-audio-stream a try and go off your demo. From the demo, if you have a window size of 5. When you start speaking do you start getting results after 5 seconds of starting speech?

[mkopcins]

Oh, so there is more thing I forgot to mention. Because the way s2t models work (that is they work in audio chunks) we needed to implement algorithm for combining the chunks together (this is what the overlappingSeconds is used for). The way this algorithm works is that we need next chunks transcription before the current chunks transcription is confirmed. So what is actually happening under the hood is that the transcription stream you are seeing is actually being returned from the previous chunk. This was a choice on our end because transcription towards the end of given audio chunk is often bad quality and we didn't want to return it to the user and then update it later. If this is something you would like to have we can add such setting in the future releases. For the latency reference please visit our docs and check the benchmark table.