❓ No onnx gpu support?

[CircuitCM]

Hi,

Just wondering is there no onnx gpu support? Would it not be any faster than jit when moving the model to CUDA with a .to() ?

This is what happened:

pip install onnxruntime-gpu

Cell In[3], line 1
----> 1 model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
      2                               model='silero_vad',
      3                               force_reload=False,
      4                               onnx=True)
      5 SAMPLE_RATE = 16000
      7 (get_speech_timestamps,
      8  save_audio,
      9  read_audio,
     10  VADIterator,
     11  collect_chunks) = utils

File ~\mambaforge\envs\AIPlayground310pip\lib\site-packages\torch\hub.py:558, in load(repo_or_dir, model, source, trust_repo, force_reload, verbose, skip_validation, *args, **kwargs)
    554 if source == 'github':
    555     repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, trust_repo, "load",
    556                                        verbose=verbose, skip_validation=skip_validation)
--> 558 model = _load_local(repo_or_dir, model, *args, **kwargs)
    559 return model

File ~\mambaforge\envs\AIPlayground310pip\lib\site-packages\torch\hub.py:587, in _load_local(hubconf_dir, model, *args, **kwargs)
    584     hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
    586     entry = _load_entry_from_hubconf(hub_module, model)
--> 587     model = entry(*args, **kwargs)
    589 return model

File ~/.cache\torch\hub\snakers4_silero-vad_master\hubconf.py:44, in silero_vad(onnx, force_onnx_cpu)
     42 model_dir = os.path.join(os.path.dirname(__file__), 'files')
     43 if onnx:
---> 44     model = OnnxWrapper(os.path.join(model_dir, 'silero_vad.onnx'), force_onnx_cpu)
     45 else:
     46     model = init_jit_model(os.path.join(model_dir, 'silero_vad.jit'))

File ~/.cache\torch\hub\snakers4_silero-vad_master\utils_vad.py:24, in OnnxWrapper.__init__(self, path, force_onnx_cpu)
     22     self.session = onnxruntime.InferenceSession(path, providers=['CPUExecutionProvider'], sess_options=opts)
     23 else:
---> 24     self.session = onnxruntime.InferenceSession(path, sess_options=opts)
     26 self.reset_states()
     27 self.sample_rates = [8000, 16000]

File ~\mambaforge\envs\AIPlayground310pip\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:396, in InferenceSession.__init__(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    394         raise fallback_error from e
    395 # Fallback is disabled. Raise the original error.
--> 396 raise e

File ~\mambaforge\envs\AIPlayground310pip\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:383, in InferenceSession.__init__(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    380 disabled_optimizers = kwargs["disabled_optimizers"] if "disabled_optimizers" in kwargs else None
    382 try:
--> 383     self._create_inference_session(providers, provider_options, disabled_optimizers)
    384 except (ValueError, RuntimeError) as e:
    385     if self._enable_fallback:

File ~\mambaforge\envs\AIPlayground310pip\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:415, in InferenceSession._create_inference_session(self, providers, provider_options, disabled_optimizers)
    413 if not providers and len(available_providers) > 1:
    414     self.disable_fallback()
--> 415     raise ValueError(
    416         f"This ORT build has {available_providers} enabled. "
    417         "Since ORT 1.9, you are required to explicitly set "
    418         "the providers parameter when instantiating InferenceSession. For example, "
    419         f"onnxruntime.InferenceSession(..., providers={available_providers}, ...)"
    420     )
    422 session_options = self._sess_options if self._sess_options else C.get_default_session_options()
    423 if self._model_path:

ValueError: This ORT build has ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'], ...)

[suki-fahmad]

@snakers4 I'm also seeing issues with onnxruntime-gpu. This is the exception I'm seeing:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running If node. Name:'If_25' Status Message: Non-zero status code returned while running FusedConv node. Name:'Conv_139' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=ml-feroz ; file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data); 

I'm using onnxruntime-gpu-1.15.1

[chazo1994]

@snakers4 I'm also seeing issues with onnxruntime-gpu. This is the exception I'm seeing:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running If node. Name:'If_25' Status Message: Non-zero status code returned while running FusedConv node. Name:'Conv_139' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=ml-feroz ; file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data); 

I'm using onnxruntime-gpu-1.15.1

I am also having this problem

[snakers4]

The VAD is not really meant to run on GPU, but maybe with a new ONNX export it will just work.

[jsoto-gladia]

you can remove the error with
opts.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_DISABLE_ALL
however the perforamnces are horrible.

[jsoto-gladia]

@snakers4 how can I do a new onnx export withtout the actual source code ?