You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's not possible to convert the Silero VAD model to work with CoreML or with other conversion tools.
To Reproduce
Steps to reproduce the behavior:
Create the following script in the root directory of this repo and pip install coremltools. Then run:
import torch
import utils_vad # from this repo
import coremltools as ct
import numpy as np
model = utils_vad.init_jit_model("files/silero_vad.jit")
model.eval()
input_features = [
ct.TensorType(name="audio", shape=torch.Size([512])),
ct.TensorType(name="sampling_rate", shape=ct.Shape((1,)), dtype=np.int64),
]
output_features = [ct.TensorType(name="output")]
coreml_model = ct.convert(
model,
inputs=input_features,
outputs=output_features,
minimum_deployment_target=ct.target.iOS15,
skip_model_load=True,
)
There are too many errors to list. I have tried to go through and comment out places where Exceptions are raised to try to get to the bottom of it, but I wasn't able to get even a broken output:
.name() does not exist on the Tensor C type when determining whether this is a quantized model
There are shape inconsistencies between different condition branches ([1,1,512] vs [1, 512])
And many more
Expected behavior
What I'm really trying to do is find a way to include silero-vad in a mobile app without having to bundle ONNX. I wasn't able to convert the .jit model to onnx myself either (I thought maybe I'd have more luck converting the resulting model to another format if it worked). I also attempted to use this tool to convert the onnx model to C but it also fails because If is not implemented there.
Environment
python collect_env.py
Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 14.4.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.28.3
Libc version: N/A
Python version: 3.12.3 | packaged by Anaconda, Inc. | (main, Apr 19 2024, 11:44:52) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Apple M2 Max
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.3.0 # it fails just the same with 2.0.0 and 2.1.0 though
[pip3] torchaudio==2.3.0
[pip3] torchvision==0.18.0
[conda] numpy 1.26.4 py312h7f4fdc5_0
[conda] numpy-base 1.26.4 py312he047099_0
[conda] pytorch 2.3.0 py3.12_0 pytorch
[conda] torchaudio 2.3.0 py312_cpu pytorch
[conda] torchvision 0.18.0 py312_cpu pytorch
Additional context
It would be really helpful to be able to modify the original Silero PyTorch model, for example to remove branching, implement the feature extractor in C directly, and so on. I'm curious whether you have considered that possibility for distribution of upcoming versions?
The text was updated successfully, but these errors were encountered:
@ephemer I just finished writing a C implementation of the v3.1 16kHz model. I'm working on it as a personal learning project, and it's very much in proof of concept stage. It probably won't build nor run anywhere but my machine atm. Having said that, if you don't mind the jank, look in my vadc repo, branch c_port_continued, this function tests the full model implementation https://github.com/IntendedConsequence/vadc/blob/b5c25db328a5fbee27a421a2d892de42bbaa3dd5/test.c#L1424
馃悰 Bug
It's not possible to convert the Silero VAD model to work with CoreML or with other conversion tools.
To Reproduce
Steps to reproduce the behavior:
Create the following script in the root directory of this repo and
pip install coremltools
. Then run:There are too many errors to list. I have tried to go through and comment out places where Exceptions are raised to try to get to the bottom of it, but I wasn't able to get even a broken output:
.name()
does not exist on the Tensor C type when determining whether this is a quantized modelExpected behavior
What I'm really trying to do is find a way to include silero-vad in a mobile app without having to bundle ONNX. I wasn't able to convert the .jit model to onnx myself either (I thought maybe I'd have more luck converting the resulting model to another format if it worked). I also attempted to use this tool to convert the onnx model to C but it also fails because
If
is not implemented there.Environment
Additional context
It would be really helpful to be able to modify the original Silero PyTorch model, for example to remove branching, implement the feature extractor in C directly, and so on. I'm curious whether you have considered that possibility for distribution of upcoming versions?
The text was updated successfully, but these errors were encountered: