Skip to content

Commit 8b92060

Browse files
authored
coreml : set convert_to="mlprogram" in convert
* coreml : skip model load in convert-whisper-to-coreml.py This commit updates the conversion process for Whisper models to use the "mlprogram" format instead of "neuralnetwork". The motivation for this change is that when using the "neuralnetwork" format the underlying model produced is based on protobuf and my understanding is that there are limitations to this format, such as sizes of strings and the complexity of the model. Currently when trying to convert larger models such as large-v3 the conversion fails but succeeds for smaller models. The "mlprogram" format is a more recent addition to CoreML and is designed to be more flexible and powerful, allowing for more complex models and larger data types. This seems to work for larger and smaller models alike and unless I'm there are considerations that I'm not aware of I think this is what we should be using moving forward. The error that is generated for large models is the following: ```console Running MIL backend_neuralnetwork pipeline: 100%|█████████| 9/9 [00:00<00:00, 35.44 passes/s] Translating MIL ==> NeuralNetwork Ops: 100%|███████████| 5641/5641 [03:31<00:00, 26.65 ops/s] Traceback (most recent call last): File "/Users/danbev/work/ai/whisper-work/models/convert-whisper-to-coreml.py", line 322, in <module> encoder = convert_encoder(hparams, encoder, quantize=args.quantize) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/danbev/work/ai/whisper-work/models/convert-whisper-to-coreml.py", line 255, in convert_encoder model = ct.convert( ^^^^^^^^^^^ File "/Users/danbev/work/ai/whisper-work/venv/lib/python3.11/site-packages/coremltools/converters/_converters_entry.py", line 635, in convert mlmodel = mil_convert( ^^^^^^^^^^^^ File "/Users/danbev/work/ai/whisper-work/venv/lib/python3.11/site-packages/coremltools/converters/mil/converter.py", line 186, in mil_convert return _mil_convert( ^^^^^^^^^^^^^ File "/Users/danbev/work/ai/whisper-work/venv/lib/python3.11/site-packages/coremltools/converters/mil/converter.py", line 245, in _mil_convert return modelClass( ^^^^^^^^^^^ File "/Users/danbev/work/ai/whisper-work/venv/lib/python3.11/site-packages/coremltools/models/model.py", line 489, in __init__ self.__proxy__, self._spec, self._framework_error = self._get_proxy_and_spec( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/danbev/work/ai/whisper-work/venv/lib/python3.11/site-packages/coremltools/models/model.py", line 550, in _get_proxy_and_spec _MLModelProxy( ValueError: basic_string ``` Refs: #3012
1 parent 7858edd commit 8b92060

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

models/convert-whisper-to-coreml.py

+4-4
Original file line numberDiff line numberDiff line change
@@ -254,10 +254,10 @@ def convert_encoder(hparams, model, quantize=False):
254254

255255
model = ct.convert(
256256
traced_model,
257-
convert_to="neuralnetwork",
257+
convert_to="mlprogram",
258258
inputs=[ct.TensorType(name="logmel_data", shape=input_shape)],
259259
outputs=[ct.TensorType(name="output")],
260-
compute_units=ct.ComputeUnit.ALL
260+
compute_units=ct.ComputeUnit.ALL,
261261
)
262262

263263
if quantize:
@@ -278,11 +278,11 @@ def convert_decoder(hparams, model, quantize=False):
278278

279279
model = ct.convert(
280280
traced_model,
281-
convert_to="neuralnetwork",
281+
convert_to="mlprogram",
282282
inputs=[
283283
ct.TensorType(name="token_data", shape=tokens_shape, dtype=int),
284284
ct.TensorType(name="audio_data", shape=audio_shape)
285-
]
285+
],
286286
)
287287

288288
if quantize:

0 commit comments

Comments
 (0)