Skip to content

Commit b648676

Browse files
JakubGoneraJakubGonera
committed
Add image segmentation docs
1 parent 19e23ac commit b648676

File tree

2 files changed

+183
-0
lines changed

2 files changed

+183
-0
lines changed

‎docs/docs/computer-vision/useImageSegmentation.md

+122
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
---
2+
title: useImageSegmentation
3+
sidebar_position: 2
4+
---
5+
6+
Semantic image segmentation, akin to image classification, tries to assign the content of the image to one of the predefined classes. However, in case of segmentation this classification is done on a per-pixel basis, so as the result the model provides an image-sized array of scores for each of the classes. React Native ExecuTorch offers a dedicated hook `useImageSegmentation`, for this task. However before you start you'll need to obtain ExecuTorch-compatible model binary.
7+
8+
:::caution
9+
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-style-transfer-candy), you can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library
10+
:::
11+
12+
## Reference
13+
14+
```typescript
15+
import {
16+
useImageSegmentation,
17+
DEEPLABV3_RESNET50,
18+
} from 'react-native-executorch';
19+
20+
const model = useImageSegmentation({
21+
modelSource: DEEPLABV3_RESNET50,
22+
});
23+
24+
const imageUri = 'file::///Users/.../cute_cat.png';
25+
26+
try {
27+
const generatedImageUrl = await model.forward(imageUri);
28+
} catch (error) {
29+
console.error(error);
30+
}
31+
```
32+
33+
### Arguments
34+
35+
**`modelSource`**
36+
A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) page.
37+
38+
### Returns
39+
40+
| Field | Type | Description |
41+
| ------------------ | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
42+
| `forward` | `(input: string, classesOfInterest?: DeeplabLabel[], resize?: boolean) => Promise<{[key in DeeplabLabel]?: number[]}>` | Executes the model's forward pass, where: <br> _ `input` can be a fetchable resource or a Base64-encoded string. <br> _ `classesOfInterest` is an optional list of `DeeplabLabel` used to indicate additional arrays of probabilities to output (see section "Running the model"). The default is an empty list. <br> _ `resize` is an optional boolean to indicate whether the output should be resized to the original image dimensions, or left in the size of the model (see section "Running the model"). The default is `false`. <br> <br> The return is a dictionary containing: <br> _ for the key `DeeplabLabel.ARGMAX` an array of integers corresponding to the most probable class for each pixel <br> \* an array of floats for each class from `classesOfInterest` corresponding to the probabilities for this class. |
43+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
44+
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
45+
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
46+
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1. |
47+
48+
## Running the model
49+
50+
To run the model, you can use the `forward` method. It accepts three arguments: a required image, an optional list of classes, and an optional flag whether to resize the output to the original dimensions.
51+
52+
- The image can be a remote URL, a local file URI, or a base64-encoded image.
53+
- The `classesOfInterest` list contains classes for which to output the full results. By default the list is empty, and only the most probable classes are returned (esentially an arg max for each pixel). Look at `DeeplabLabel` enum for possible classes.
54+
- The `resize` flag says whether the output will be rescaled back to the size of the image you put in. The default is `false`. The model runs inference on a scaled (probably smaller) version of your image (224x224 for the `DEEPLABV3_RESNET50`). If you choose to resize, the output will be `number[]` of size `width * height` of your original image.
55+
56+
:::caution
57+
Setting `resize` to true will make `forward` slower
58+
:::
59+
60+
`forward` returns a promise which can resolve either to an error or a dictionary containing number arrays with size depending on `resize`:
61+
62+
- For the key `DeeplabLabel.ARGMAX` the array contains for each pixel an integer corresponding to the class with the highest probability.
63+
- For every other key from `DeeplabLabel`, if the label was included in `classesOfInterest` the dictionary will contain an array of floats corresponding to the probability of this class for every pixel.
64+
65+
## Example
66+
67+
```typescript
68+
function App(){
69+
const model = useImageSegmentation(
70+
modelSource: DEEPLABV3_RESNET50,
71+
);
72+
73+
...
74+
const imageUri = 'file::///Users/.../cute_cat.png';
75+
76+
try{
77+
const outputDict = await model.forward(imageUri, [DeeplabLabel.CAT], true);
78+
}catch(error){
79+
console.error(error);
80+
}
81+
...
82+
}
83+
```
84+
85+
## Supported models
86+
87+
| Model | Number of classes | Class list |
88+
| ------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ---------- |
89+
| [deeplabv3_resnet50](https://pytorch.org/vision/0.20/models/generated/torchvision.models.segmentation.deeplabv3_resnet50.html) | 21 | [TODO]() |
90+
91+
## Benchmarks
92+
93+
### Model size
94+
95+
| Model | XNNPACK [MB] | Core ML [MB] |
96+
| ---------------------------- | ------------ | ------------ |
97+
| STYLE_TRANSFER_CANDY | 6.78 | 5.22 |
98+
| STYLE_TRANSFER_MOSAIC | 6.78 | 5.22 |
99+
| STYLE_TRANSFER_UDNIE | 6.78 | 5.22 |
100+
| STYLE_TRANSFER_RAIN_PRINCESS | 6.78 | 5.22 |
101+
102+
### Memory usage
103+
104+
| Model | Android (XNNPACK) [MB] | iOS (Core ML) [MB] |
105+
| ---------------------------- | ---------------------- | ------------------ |
106+
| STYLE_TRANSFER_CANDY | 950 | 350 |
107+
| STYLE_TRANSFER_MOSAIC | 950 | 350 |
108+
| STYLE_TRANSFER_UDNIE | 950 | 350 |
109+
| STYLE_TRANSFER_RAIN_PRINCESS | 950 | 350 |
110+
111+
### Inference time
112+
113+
:::warning warning
114+
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
115+
:::
116+
117+
| Model | iPhone 16 Pro (Core ML) [ms] | iPhone 13 Pro (Core ML) [ms] | iPhone SE 3 (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
118+
| ---------------------------- | ---------------------------- | ---------------------------- | -------------------------- | --------------------------------- | ------------------------- |
119+
| STYLE_TRANSFER_CANDY | 450 | 600 | 750 | 1650 | 1800 |
120+
| STYLE_TRANSFER_MOSAIC | 450 | 600 | 750 | 1650 | 1800 |
121+
| STYLE_TRANSFER_UDNIE | 450 | 600 | 750 | 1650 | 1800 |
122+
| STYLE_TRANSFER_RAIN_PRINCESS | 450 | 600 | 750 | 1650 | 1800 |

‎docs/docs/hookless-api/ImageSegmentationModule.md

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
title: ImageSegmentationModule
3+
sidebar_position: 4
4+
---
5+
6+
Hookless implementation of the [useImageSegmentation](../computer-vision/useImageSegmentation.md) hook.
7+
8+
## Reference
9+
10+
```typescript
11+
import {
12+
ImageSegmentationModule,
13+
DEEPLABV3_RESNET50,
14+
} from 'react-native-executorch';
15+
16+
const imageUri = 'path/to/image.png';
17+
18+
// Loading the model
19+
await ImageSegmentationModule.load(DEEPLABV3_RESNET50);
20+
21+
// Running the model
22+
const outputDict = await StyleTransferModule.forward(imageUri);
23+
```
24+
25+
### Methods
26+
27+
| Method | Type | Description |
28+
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
29+
| `load` | `(modelSource: ResourceSource): Promise<void>` | Loads the model, where `modelSource` is a string that specifies the location of the model binary. |
30+
| `forward` | `(input: string, classesOfInterest?: DeeplabLabel[], resize?: boolean) => Promise<{[key in DeeplabLabel]?: number[]}>` | Executes the model's forward pass, where : <br> _ `input` can be a fetchable resource or a Base64-encoded string. <br> _ `classesOfInterest` is an optional list of `DeeplabLabel` used to indicate additional arrays of probabilities to output (see section "Running the model"). The default is an empty list. <br> _ `resize` is an optional boolean to indicate whether the output should be resized to the original image dimensions, or left in the size of the model (see section "Running the model"). The default is `false`. <br> <br> The return is a dictionary containing: <br> _ for the key `DeeplabLabel.ARGMAX` an array of integers corresponding to the most probable class for each pixel <br> \* an array of floats for each class from `classesOfInterest` corresponding to the probabilities for this class. |
31+
| `onDownloadProgress` | `(callback: (downloadProgress: number) => void): any` | Subscribe to the download progress event. |
32+
33+
<details>
34+
<summary>Type definitions</summary>
35+
36+
```typescript
37+
type ResourceSource = string | number;
38+
```
39+
40+
</details>
41+
42+
## Loading the model
43+
44+
To load the model, use the `load` method. It accepts the `modelSource` which is a string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) page. This method returns a promise, which can resolve to an error or void.
45+
46+
## Running the model
47+
48+
To run the model, you can use the `forward` method. It accepts three arguments: a required image, an optional list of classes, and an optional flag whether to resize the output to the original dimensions.
49+
50+
- The image can be a remote URL, a local file URI, or a base64-encoded image.
51+
- The `classesOfInterest` list contains classes for which to output the full results. By default the list is empty, and only the most probable classes are returned (esentially an arg max for each pixel). Look at `DeeplabLabel` enum for possible classes.
52+
- The `resize` flag says whether the output will be rescaled back to the size of the image you put in. The default is `false`. The model runs inference on a scaled (probably smaller) version of your image (224x224 for the `DEEPLABV3_RESNET50`). If you choose to resize, the output will be `number[]` of size `width * height` of your original image.
53+
54+
:::caution
55+
Setting `resize` to true will make `forward` slower
56+
:::
57+
58+
`forward` returns a promise which can resolve either to an error or a dictionary containing number arrays with size depending on `resize`:
59+
60+
- For the key `DeeplabLabel.ARGMAX` the array contains for each pixel an integer corresponding to the class with the highest probability.
61+
- For every other key from `DeeplabLabel`, if the label was included in `classesOfInterest` the dictionary will contain an array of floats corresponding to the probability of this class for every pixel.

0 commit comments

Comments
 (0)