onnx inference speed

[xIaott-s]

Hi, I export the ecapa-tdnn model to onnx and use onnxruntime to inference. But the inference speed is slower than pytorch. Does anybody konw why?

[aheba]

@sty1992928 , as far as I know,
Generally, using ONNX + TensorRT improve by high range the performance,

are you using this project ?
https://github.com/NVIDIA-AI-IOT/torch2trt

[vadimkantorov]

Onnxruntime has some unsolved perf problems with conv1d: microsoft/onnxruntime#7212 microsoft/onnxruntime#8513

We found only version 1.8.0 is somewhat performant (usually these issues are about bad configuration of cudnn / cudnn benchmarking workspace size). Please also report to onnxruntime to bump these issues in priority

[xIaott-s]

are you using this project ?

I use cpu only，is this as expected？

[xIaott-s]

Onnxruntime has some unsolved perf problems with conv1d: microsoft/onnxruntime#7212 microsoft/onnxruntime#8513

We found only version 1.8.0 is somewhat performant (usually these issues are about bad configuration of cudnn / cudnn benchmarking workspace size). Please also report to onnxruntime to bump these issues in priority

I use cpu only, does cpu mode has the same perf problems?

[vadimkantorov]

Those issues are unrelated to CPU, so likely not...

[xIaott-s]

Those issues are unrelated to CPU, so likely not...

Thank you anyway！The Int8 quant works，so my problem is temporarily gone.

[xle97]

Hi, I export the ecapa-tdnn model to onnx and use onnxruntime to inference. But the inference speed is slower than pytorch. Does anybody konw why?

hi，May I ask how you exported onnx？

[aheba]

Hi you can use torch.onnx.export to do the onnx export of torch module;
Basically you will need for the forward + a tracing example..

[marziye-A]

Hi, I export the ecapa-tdnn model to onnx and use onnxruntime to inference. But the inference speed is slower than pytorch. Does anybody konw why?

dear @xIaott-s, can you please share your code on how you have converted your model to onnx and quantized it?