Support React Native #118

hans00 · 2023-05-19T05:28:07Z

Make it support React Native.

And I made a example for it.
https://github.com/hans00/react-native-transformers-example

~~On Android need some library patch.~~

TODO:

~~Check models are works fine~~
~~Research more efficiently image processing~~
Merge v3
Check everything are works fine on v3

xenova · 2023-05-19T05:33:57Z

Woah nice! There are a lot of people who are interested in doing stuff like this! Looking forward to reviewing this when you're ready! cc @pcuenca

package.json

jhen0409 · 2023-05-20T02:05:46Z

I'm investigating a performance issue with Whisper tiny.en, and it looks like the performance is not as expected in the example.

I quickly add log for onnxruntime-react-native/lib/backend.ts, this is the result:

# Encode result
 LOG  ONNX session run finished 767 ms

# Decode result
 LOG  ONNX session run finished 4518 ms
# ... 4 runs ...
 LOG  ONNX session run finished 4856 ms
 LOG  Time: 40016
 LOG  Result: {"text": " (buzzing)"}

Click to expand

const t0 = performance.now()
// this.#inferenceSession === NativeModules.Onnxruntime
const results: Binding.ReturnType = await this.#inferenceSession.run(this.#key, input, outputNames, options);
const output = this.decodeReturnType(results);
console.log('ONNX session run finished', performance.now() - t0)

Also see logs of native part (added log to OnnxruntimeModule.java), it is significantly less than the time taken by JS part:

# Encode result
09:47:59.203 ONNXRUNTIME_RN run() is finished, time: 273 ms

# Decode result
09:48:00.280 ONNXRUNTIME_RN run() is finished, time: 339 ms
# ... 4 runs ...
09:48:23.807 ONNXRUNTIME_RN run() is finished, time: 541 ms

~~I think this problem may come from blocking caused by native module passing too large data. I think request/help onnxruntime for migrate to JSI module may solve this problem.~~

EDIT: It seems the logger have some bugs leads me to think that the problem is from the native bridge. I add timeout await to the decodeReturnType call and found the issue is from this function.

jpcoder2 · 2025-03-19T01:39:45Z

@hans00 , I'v seen a lot of commits from you on this lately. Thanks for that!

Were you able to get this working? Do you know if it will be merged into the main project soon?

hans00 · 2025-03-25T06:04:44Z

@xenova ping

xenova · 2025-04-13T02:06:17Z

Hi @hans00! 👋 Thanks so much for your work here - it will be very impactful.

I'm finally able to commit more time to this, as I now have the ability to do mobile development in my environment. My main concern with the PR in its current state is the additional dependencies that it introduces. Ideally, the only new dependency that would be introduced is onnxruntime-react-native. For things like image processing, if sharp can't handle it, then I think we should let the user implement the image reading themselves (at least to start off with).

What do you think?

hans00 · 2025-04-14T10:57:03Z

For things like image processing, if sharp can't handle it, then I think we should let the user implement the image reading themselves (at least to start off with).

What do you think?

Ok, I had removed JS codecs for RN.
For better performance, our project are uses OffscreenCanvas polyfill.

For FS access, native-universal-fs serves merely as a generic wrapper for the FS modules of React Native or Expo without directly depending on the modules themselves.
This allows for type checking, which is preferable to aliasing fs as the FS module and thereby only supporting a specific FS module.

simonwh · 2025-05-27T15:24:41Z

This is absolutely brilliant work @hans00, big props for your relentlessness in over almost 2 years 🙇‍♂️ I hope this will get the attention it deserves from @xenova and the team and get merged into main soon!

TowhidKashem · 2025-07-04T08:18:23Z

hey @hans00 , sorry to bother you but I've been trying without success to get your changes working for the past day and a half and thought I'd ask... I installed @fugood/transformers along with all the polyfills and such and followed ur example repo.

I manually placed a model in my app's application folder:

https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english/tree/main

the contents are in a folder called distilbert-base-uncased-finetuned-sst-2-english and contains the exact repo's contents above.

When I try to use it like:

import { cacheDirectory, documentDirectory } from 'expo-file-system';

env.localModelPath = documentDirectory;
env.cacheDir = cacheDirectory;
env.allowRemoteModels = false;

export const sentimentAnalysis = async (text: string) => {
  const classifier = await pipeline(
    'sentiment-analysis',
    'distilbert-base-uncased-finetuned-sst-2-english',
    {
      // progress_callback: (progress) => console.info('progress', progress)
    }
  );
  const result = await classifier(text); // [{'label': 'POSITIVE', 'score': 0.999817686}]
  return result;
};

export const sentimentAnalysisAsString = async (text: string) =>
  sentimentAnalysis(text).then((result) => JSON.stringify(result, null, 2));

It gives me these 3 warnings:

WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/tokenizer_config.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/config.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"

It seems it might have come from the native-universal-fs lib you're using, I tried replacing those instances with expo-file-system directly and using patch-package to update @fugood/transformers but the errors remain.

Any idea what could be the fix?

axe-me · 2025-07-04T08:35:28Z

@TowhidKashem I ran into same issue. I found it was because the metro reads the exports field of the package then loaded the browser version build. You can use my fork for now: https://www.npmjs.com/package/@axe-dev/transformers (I couldn't guarantee that I won't add my changes in there) or you can figure out some ways to load the build that bundled the universal fs package.

VikingLichens · 2025-07-04T10:11:25Z

hey @hans00 , sorry to bother you but I've been trying without success to get your changes working for the past day and a half and thought I'd ask... I installed @fugood/transformers along with all the polyfills and such and followed ur example repo.

I manually placed a model in my app's application folder:

https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english/tree/main

the contents are in a folder called distilbert-base-uncased-finetuned-sst-2-english and contains the exact repo's contents above.

When I try to use it like:

import { cacheDirectory, documentDirectory } from 'expo-file-system';

env.localModelPath = documentDirectory;
env.cacheDir = cacheDirectory;
env.allowRemoteModels = false;

export const sentimentAnalysis = async (text: string) => {
  const classifier = await pipeline(
    'sentiment-analysis',
    'distilbert-base-uncased-finetuned-sst-2-english',
    {
      // progress_callback: (progress) => console.info('progress', progress)
    }
  );
  const result = await classifier(text); // [{'label': 'POSITIVE', 'score': 0.999817686}]
  return result;
};

export const sentimentAnalysisAsString = async (text: string) =>
  sentimentAnalysis(text).then((result) => JSON.stringify(result, null, 2));

It gives me these 3 warnings:

WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/tokenizer_config.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/config.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"

It seems it might have come from the native-universal-fs lib you're using, I tried replacing those instances with expo-file-system directly and using patch-package to update @fugood/transformers but the errors remain.

Any idea what could be the fix?

I had same issue in my recent trials...

Not working directly with Expo mybigday/native-universal-fs#1

TowhidKashem · 2025-07-05T06:09:42Z

hey @hans00 , sorry to bother you but I've been trying without success to get your changes working for the past day and a half and thought I'd ask... I installed @fugood/transformers along with all the polyfills and such and followed ur example repo.
I manually placed a model in my app's application folder:
https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english/tree/main
the contents are in a folder called distilbert-base-uncased-finetuned-sst-2-english and contains the exact repo's contents above.
When I try to use it like:
import { cacheDirectory, documentDirectory } from 'expo-file-system';

env.localModelPath = documentDirectory;
env.cacheDir = cacheDirectory;
env.allowRemoteModels = false;

export const sentimentAnalysis = async (text: string) => {
  const classifier = await pipeline(
    'sentiment-analysis',
    'distilbert-base-uncased-finetuned-sst-2-english',
    {
      // progress_callback: (progress) => console.info('progress', progress)
    }
  );
  const result = await classifier(text); // [{'label': 'POSITIVE', 'score': 0.999817686}]
  return result;
};

export const sentimentAnalysisAsString = async (text: string) =>
  sentimentAnalysis(text).then((result) => JSON.stringify(result, null, 2));
It gives me these 3 warnings:
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/tokenizer_config.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
WARN  Unable to load from local path "file:///Users/redacted/Library/Developer/CoreSimulator/Devices/B1A4461B-D329-482B-BAE5-991F910EFCCD/data/Containers/Data/Application/78CFCE04-ED0E-448D-BA36-402E2C0C1613/Documents/distilbert-base-uncased-finetuned-sst-2-english/config.json": "TypeError: native_universal_fs__WEBPACK_IMPORTED_MODULE_2__.exists is not a function (it is undefined)"
It seems it might have come from the native-universal-fs lib you're using, I tried replacing those instances with expo-file-system directly and using patch-package to update @fugood/transformers but the errors remain.
Any idea what could be the fix?
I had same issue in my recent trials...

Not working directly with Expo mybigday/native-universal-fs#1

I was able to get onnx models working using just onnxruntime-react-native, that lib also gave that same error which I was able to suppress using a stub:

onnx-web.stub.ts

export default {};
export const InferenceSession = {};
export const Tensor = {};
export const env = {};

babel.config.js:

    plugins: [
      [
        'module-resolver',
        {
          root: ['./'],
          alias: {
            'onnxruntime-web': './onnx-web.stub.ts'
          }
        }

this gets rid of the webpack error and allows u to use onnx models. But the experience isn't as easy as using transformers.js so that would still be much more preferred. Without transformers.js you have to figure out unique tokenizers for every model which is a real pain in the butt..

TowhidKashem · 2025-07-05T06:12:14Z

@TowhidKashem I ran into same issue. I found it was because the metro reads the exports field of the package then loaded the browser version build. You can use my fork for now: https://www.npmjs.com/package/@axe-dev/transformers (I couldn't guarantee that I won't add my changes in there) or you can figure out some ways to load the build that bundled the universal fs package.

Thanks, is this the right repo:

https://github.com/axe-me/transformers.js/tree/main

and which branch should I be forking?

axe-me · 2025-07-05T06:39:03Z

@TowhidKashem same as the fugood one, merge branch.
Or you can just install it : npm i @axe-dev/transformers

hans00 · 2025-07-05T16:43:33Z

It seems it might have come from the native-universal-fs lib you're using, I tried replacing those instances with expo-file-system directly and using patch-package to update @fugood/transformers but the errors remain.
Any idea what could be the fix?

I had same issue in my recent trials...

Not working directly with Expo mybigday/native-universal-fs#1

This should fixed.
But currently uses webpack bundle to solve package alias, so that could not install from source.

hans00 and others added 17 commits May 17, 2023 11:50

Support for ReactNative

1aefcb6

Correct module import

38acc1e

Update package-lock

0d7faf0

fix errors

c1da2c8

fix error

bb550f1

Merge branch 'xenova:main' into main

35d8f18

Prevent fallback WASM on RN

83cd1e6

Let native side load binary file, instead load as buffer

f64e61b

Fix 0 size tensor

85d75ef

Correct version

392dc67

Constantly fetch arraybuffer

c9cada9

Replace Uint8Array as Buffer on RN

20f128f

Revert version

891a0a4

Merge native codes

fd302ae

Fix test

37e89cf

Disable image-encode and image-decode for web

9ee7504

Sync code

91ecc47

hans00 and others added 5 commits May 19, 2023 14:52

Use RNFS.downloadFile to avoid OOM on large model file.

17bc3e8

Correct download progress order

58c2468

Fix bug

52be4db

Merge branch 'xenova:main' into main

e0683b3

Fix error

362a0c7

xenova reviewed May 19, 2023

View reviewed changes

package.json Outdated Show resolved Hide resolved

hans00 added 3 commits May 19, 2023 19:37

Use interpolate_data instead resize-image-data

c633a2d

Correct params

798edec

Support react-native-gcanvas to improve image parse performance

8c66b31

jhen0409 mentioned this pull request May 20, 2023

[Experimental] Use blob for decode result instead of base64 hans00/react-native-transformers-example#1

Closed

Correct algorithm impl

874340c

hans00 and others added 9 commits February 11, 2025 14:41

Manually mock react-native for test

8f4b0e3

Fix test or non-RN env work

7285456

Use fetch under globalThis

6f01fc0

Merge branch 'huggingface:main' into merge

6a6c5b1

Merge branch 'main' into merge

578fd5b

Download to cache dir if useFSCache

07ccf2d

Fix download model

da6eafa

Remove mistake

4c1da9b

Fix model load

2486900

hans00 added 2 commits April 3, 2025 16:00

Merge branch 'main' into merge

cbbf58e

Fix download target path

1db6417

Remove image-codecs, let user implement them-self

c746cf9

daviddaytw mentioned this pull request Apr 18, 2025

Support for React Native in huggingface/transformers.js daviddaytw/react-native-transformers#10

Open

hans00 added 2 commits April 22, 2025 14:50

Merge branch 'main' into merge

9f3f32c

Merge branch 'main' into merge

be92e91

hans00 added 2 commits July 5, 2025 23:06

Merge branch 'main' into merge

c3e8ba0

Use webpack to bundle for React Native or Expo

d0c157b

Fix RN is not support import.meta

e2eb7ca

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support React Native #118

Support React Native #118

hans00 commented May 19, 2023 •

edited

Loading

Uh oh!

xenova commented May 19, 2023 •

edited

Loading

Uh oh!

Uh oh!

jhen0409 commented May 20, 2023 •

edited

Loading

Uh oh!

jpcoder2 commented Mar 19, 2025

Uh oh!

hans00 commented Mar 25, 2025

Uh oh!

xenova commented Apr 13, 2025 •

edited

Loading

Uh oh!

hans00 commented Apr 14, 2025 •

edited

Loading

Uh oh!

simonwh commented May 27, 2025

Uh oh!

TowhidKashem commented Jul 4, 2025

Uh oh!

axe-me commented Jul 4, 2025

Uh oh!

VikingLichens commented Jul 4, 2025

Uh oh!

TowhidKashem commented Jul 5, 2025

Uh oh!

TowhidKashem commented Jul 5, 2025

Uh oh!

axe-me commented Jul 5, 2025

Uh oh!

hans00 commented Jul 5, 2025

Uh oh!

Uh oh!

Support React Native #118

Are you sure you want to change the base?

Support React Native #118

Conversation

hans00 commented May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xenova commented May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jhen0409 commented May 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jpcoder2 commented Mar 19, 2025

Uh oh!

hans00 commented Mar 25, 2025

Uh oh!

xenova commented Apr 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hans00 commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonwh commented May 27, 2025

Uh oh!

TowhidKashem commented Jul 4, 2025

Uh oh!

axe-me commented Jul 4, 2025

Uh oh!

VikingLichens commented Jul 4, 2025

Uh oh!

TowhidKashem commented Jul 5, 2025

Uh oh!

TowhidKashem commented Jul 5, 2025

Uh oh!

axe-me commented Jul 5, 2025

Uh oh!

hans00 commented Jul 5, 2025

Uh oh!

Uh oh!

hans00 commented May 19, 2023 •

edited

Loading

xenova commented May 19, 2023 •

edited

Loading

jhen0409 commented May 20, 2023 •

edited

Loading

xenova commented Apr 13, 2025 •

edited

Loading

hans00 commented Apr 14, 2025 •

edited

Loading