5x pytorch performance increase on CPU, 1 thread

[IntendedConsequence]

tl;dr: Smarter batching without splitting the whole audio. Use batch inference on steps up until and including the encoder, then use batch size of 1 for lstm and decoder, because the last two parts are the only ones that depend on state.

All that is without affecting precision at all, as would be the case with regular batching, where one would have to split the entire audio in multiple independent slices, where each slice would be processed using its own independent state. But if batching is only used on the graph up to encoder, you can still process the entire audio with a single state.

Longer version:
When I was looking inside the jit and onnx model internals, I noticed that the majority of the computation the model performs during inference does not depend on state (hn and cn inputs to lstm).

The model consists of roughly 6 steps:

• feature_extractor
• adaptive_normalization
• first_layer
• encoder
• lstm
• decoder

First 4 of those depend only on audio data, and only at the lstm step the state variables hn and cn are actually used and updated. This means that first 4 steps are easy to isolate and parallelize, and only the last 2 steps need to be computed sequentially.
Note: technically the decoder can also be batched, resulting in more speedups, but I didn't test that

While this is good news for GPU and multithreaded inference, I did not expect a 5x performance increase on a single thread.

Version: Silero VAD v3.1, 16kHz. Restored python code from .jit file (for implementing the model architecture in C/ggml)
Sample count: 1536
Pytorch version: 2.1-cpu

torch.set_grad_enabled(False)
torch.set_num_threads(1)
torch.set_num_interop_threads(1)

Batch size for first 4 steps: 32
Batch size for lstm and decoder: 1

Result: 600s of audio data is processed in 2-3 seconds.
And this is just using my restored pytorch silero code without any TorchScript, no jit.script or jit.trace at all.

Running the official v3.1 silero_vad.jit, batch_size=1 on the same 600s of audio data takes 10.4 seconds.

Batch size above 32 on my cpu doesn't result in much improvement.

I didn't look at V4 internals closely, but if the lstm is still near the bottom of the graph, the preceding steps can probably benefit from batching too.
I also didn't test what effect this kind of batching have on onnx model inference, and I didn't conduct any GPU/CUDA tests.

[IntendedConsequence]

UPD: onnx < 1s
I did a very quick and dirty test with exported onnx of the restored pytorch code with hardcoded batch size 32 for the non-lstm steps, and the same 600s of audio data are being processed in a little less than a second. CPU, 1 thread, and using python's onnxruntime bindings.
If these results aren't a fluke, it would make it 600x realtime processing speed. It's also more than twice as fast as using the official .onnx v3.1 model from pure C, which processes the same audio data in about 2s.

P.S. Admittedly this was a very fast proof of concept test, so I may have screwed something up, even though result probabilities are all within 2e-6 compared to the reference probabilities.

[IntendedConsequence]

UPD: onnx CPU 800x up to 1240x speed
I've tweaked the onnx model a bit since, it supports dynamic batching now.

These were constant in my tests:

• Sample count per chunk = 1536
• Batch size = 96
• SetInterOpNumThreads = 1

With SetIntraOpNumThreads of 1, through onnxruntime C api, I could get silero model process 5071 seconds of PCM 16-bit 16kHz audio in just under 6.4 seconds, giving about 800x processing speed.

Increasing SetIntraOpNumThreads to 4 boosts processing speed up to 1240x, taking less than 4.1 seconds to process the same audio.

This is a best case scenario though, when the audio was decoded, mixed and resampled into the required format ahead of time. If I decode opus-encoded audio with ffmpeg and pipe the output to the vad utility, the speed is bottlenecked by the decode speed and falls to ~340x. Also do note, these measurements were made on AMD 5950x.

I've switched to this modified onnx model and been testing it in daily use for months, and so far it seems to run just fine.

If anyone wants to test this modified onnx model, I'm attaching it. No guarantees though, not even that it'll work on your setup, YMMV =)

The input and output names should be the same, the only difference is that "input" shape is now [B, 1536], so 1536 samples is hardcoded because this is the sample count that I was using historically, and "output"'s first axis will also correspond to B. I never tried other sample input counts though, and I don't know if restoring the original variable sample count support would hurt performance.

Also the modified onnx model is onnx V8, while the original silero 3.1 is onnx v6. This is just what the default torch 2.1.1 exported, not a deliberate choice. I also ran it through an onnx simplifier and optimizer at https://convertmodel.com

silero_restored_v3.1_16k_v2_dyn-sim.zip

[dgoryeo]

@IntendedConsequence , any guidance / sample code on how the silero model can be used? Thanks!

[IntendedConsequence]

@dgoryeo silero repository already has example python code for onnx inference. The 3.1 model has different input names from v4.0 though, so you'd have to look at the 3.1 tag, here:

silero-vad/utils_vad.py

Line 10 in 8ebaf13

class OnnxWrapper():

If you're interested in using it via onnxruntime's C api however, well, you can look in my vadc repository, but it's highly experimental and WIP, I doubt it will even compile on any other machine in its current state. So it probably won't help you if you aren't comfortable reading C 🤷‍♂️