v1.5.5

[ggerganov]

Overview

Many small incremental updates + Token level timestamps with DTW by @denersc in #1485
Feedback is welcome!

Full Changelog: v1.5.4...v1.5.5

What's Changed

• server : fix server temperature + add temperature_inc by @ggerganov in server : fix server temperature + add temperature_inc #1729
• main : add cli option to disable system prints by @ggerganov in main : add cli option to disable system prints #1740
• server: add request path by @eschmidbauer in server: add request path #1741
• Optional Piper TTS support for talk-llama example. by @RhinoDevel in Optional Piper TTS support for talk-llama example. #1749
• fix/1748 by @nank1ro in fix/1748 #1750
• Don't compute timestamps when not printing them. by @ghindle in Don't compute timestamps when not printing them. #1755
• Add more parameters to server api by @ghindle in Add more parameters to server api #1754
• Add SetInitialPrompt method to go bindings by @blib in Add SetInitialPrompt method to go bindings #1753
• ggml : fix 32-bit ARM compat for IQ2_XS by @ggerganov in ggml : fix 32-bit ARM compat for IQ2_XS #1758
• refactor: get all scripts to be POSIX Compliant by @sonphantrung in refactor: get all scripts to be POSIX Compliant #1725
• whisper : load the model into multiple buffers of max size 1GB by @ggerganov in whisper : load the model into multiple buffers of max size 1GB #1763
• rebase against your -np changes (thx) and add better python file to be used on the command line or as library by @contractorwolf in rebase against your -np changes (thx) and add better python file to be used on the command line or as library #1744
• examples/talk-llama: Add optional commandline parameter to set the bot name. by @RhinoDevel in examples/talk-llama: Add optional commandline parameter to set the bot name. #1764
• server : fix building and simplify lib deps on Windows by @przemoc in server : fix building and simplify lib deps on Windows #1772
• talk-llama: optional wake-up command and audio confirmation by @Rakksor in talk-llama: optional wake-up command and audio confirmation #1765
• examples/server: implement "verbose_json" format with token details by @rmmh in examples/server: implement "verbose_json" format with token details #1781
• whisper.android: Return output from benchmarks by @luciferous in whisper.android: Return output from benchmarks #1785
• libwhisper.so should be position independent by @trixirt in libwhisper.so should be position independent #1792
• Docs: try to make model options / model install methods clearer by @mrienstra in Docs: try to make model options / model install methods clearer #1806
• common : fix input buffer check by @ggerganov in common : fix input buffer check #1812
• Update Makefile by @jwijffels in Update Makefile #1813
• Add fields to verbose_json response and show examples on the home page by @JacobLinCool in Add fields to verbose_json response and show examples on the home page #1802
• common: fix wav buffer detection by @JacobLinCool in common: fix wav buffer detection #1819
• Add macOS deployment target option to Makefile by @didzis in Add macOS deployment target option to Makefile #1839
• Expose CUDA device setting in public API by @didzis in Expose CUDA device setting in public API #1840
• whisper.android: How to build with CLBlast by @luciferous in whisper.android: How to build with CLBlast #1809
• server: Allow CORS request with authorization headers by @valenting in server: Allow CORS request with authorization headers #1850
• Embed Metal library source into compiled binary by @didzis in Embed Metal library source into compiled binary #1842
• added audio_ctx argument to main and server examples by @dscripka in added audio_ctx argument to main and server examples #1857
• whisper : fix external encoder by @ggerganov in whisper : fix external encoder #1860
• swift : package no longer use ggml dependency by @ggerganov in swift : package no longer use ggml dependency #1861
• fix openvino setup docs by @jumpers775 in fix openvino setup docs #1874
• clean up common code in examples by @felrock in clean up common code in examples #1871
• main : check if input files exist before proceeding by @Theldus in main : check if input files exist before proceeding #1872
• Linking issue fix via Makefile when CUBLAS enabled in the WSL Compile failure with CUBLAS enabled in the WSL #1876 by @lbluep in Linking issue fix via Makefile when CUBLAS enabled in the WSL #1876 #1878
• main : fix file existence check in main.cpp by @Theldus in main : fix file existence check in main.cpp #1889
• openvino : fix convert-whisper-to-openvino.py for v2023.0.0 (Conversion from Whisper to OpenVino failed #1870) by @st-gr in openvino : fix convert-whisper-to-openvino.py for v2023.0.0 (#1870) #1890
• ggml : 32-bit arm compat by @ggerganov in ggml : 32-bit arm compat #1891
• Add SYCL logic in whisper by @abhilash1910 in Add SYCL logic in whisper #1863
• talk and talk-llama: Pass text_to_speak as a file by @tamo in talk and talk-llama: Pass text_to_speak as a file #1865
• Stream.wasm: Fix invalid memory access when no segments are returned by @Andrews54757 in Stream.wasm: Fix invalid memory access when no segments are returned #1902
• Update README to Recommend MacOS Sonoma for Core ML to avoid hallucination by @gavin1818 in Update README to Recommend MacOS Sonoma for Core ML to avoid hallucination #1917
• Add library versioning by @kenneth-ge in Add library versioning #1352
• Fix SF(segment fault) issue in Android JNI by @zhouwg in Fix SF(segment fault) issue in Android JNI #1929
• Fix typo in source file whisper.cpp by @zhouwg in Fix typo in source file whisper.cpp #1925
• bench:fix typo by @zhouwg in bench:fix typo #1933
• Auto lowercase language parameter by @F1L1Pv2 in Auto lowercase language parameter #1928
• ggml : try fix 32-bit arm compat by @ggerganov in ggml : try fix 32-bit arm compat #1938
• whisper : make beam candidate sort more stable by @josharian in whisper : make beam candidate sort more stable #1943
• bindings/go : add linker flags to make metal work by @josharian in bindings/go : add linker flags to make metal work #1944
• whisper : improve beam search candidate diversity by @josharian in whisper : improve beam search candidate diversity #1947
• whisper : document whisper_batch.n_seq_id by @josharian in whisper : document whisper_batch.n_seq_id #1942
• Rename --audio-context to --audio-ctx, as per help text by @joliss in Rename --audio-context to --audio-ctx, as per help text #1953
• [DRAFT] Token level timestamps with DTW (Implement word-level timestamps approach proposed by OpenAI #375) by @denersc in [DRAFT] Token level timestamps with DTW (#375)  #1485
• Fedora dependencies needed (SDL2) by @Man2Dev in Fedora dependencies needed (SDL2) #1970
• libcuda.so.1 in PATH in Docker Container by @tiagofassoni in libcuda.so.1 in PATH in Docker Container #1966
• ruby : fix build by @ggerganov in ruby : fix build #1980
• Improve support for distil-large-v3 by @sanchit-gandhi in Improve support for distil-large-v3 #1982
• whisper : improve handling of prompts by @ggerganov in whisper : improve handling of prompts #1981
• sync : ggml by @ggerganov in sync : ggml #2001
• Implemented command-style grammar in the main example. by @ulatekh in Implemented command-style grammar in the main example. #1998
• Use pkg-config for OpenBLAS by @przemoc in Use pkg-config for OpenBLAS #1778
• ci : add building in MSYS2 environments (Windows) by @przemoc in ci : add building in MSYS2 environments (Windows) #1994
• Support CUDA versions < 11.1 by @primenko-v in Support CUDA versions < 11.1 #2020
• Create solution folders in the CMake build by @ulatekh in Create solution folders in the CMake build #2004
• Allow a regular expression to describe tokens to suppress by @ulatekh in Allow a regular expression to describe tokens to suppress #1997
• "main" example now allows a response-file as the sole parameter by @ulatekh in "main" example now allows a response-file as the sole parameter #2019
• Support for CPU BLAS build via Intel MKL by @slashlib in Support for CPU BLAS build via Intel MKL #2024
• Set stdin to binary mode on Windows. Fixes Piping to stdin is failing on Windows #2023 by @rotemdan in Set stdin to binary mode on Windows. Fixes #2023 #2025
• Fix file-handle leak in read_wav() by @ulatekh in Fix file-handle leak in read_wav() #2026
• Fix DTW memory access by @bradmurray-dt in Fix DTW memory access #2012
• whisper: update grammar-parser.cpp by @eltociear in whisper: update grammar-parser.cpp #2058
• fix missing reference to "model" variable in actual shell command run in whisper.nvim by @sixcircuit in fix missing reference to "model" variable in actual shell command run in whisper.nvim #2049
• build : detect AVX512 in Makefile, add AVX512 option in CMake by @didzis in build : detect AVX512 in Makefile, add AVX512 option in CMake #2043
• feature/no timestamps node by @pprobst in feature/no timestamps node #2048
• Update embedded Metal library generation process to include dependency by @didzis in Update embedded Metal library generation process to include dependency #2045
• server.cpp: add dtw by @eschmidbauer in server.cpp: add dtw #2044

New Contributors

• @eschmidbauer made their first contribution in server: add request path #1741
• @RhinoDevel made their first contribution in Optional Piper TTS support for talk-llama example. #1749
• @nank1ro made their first contribution in fix/1748 #1750
• @ghindle made their first contribution in Don't compute timestamps when not printing them. #1755
• @blib made their first contribution in Add SetInitialPrompt method to go bindings #1753
• @sonphantrung made their first contribution in refactor: get all scripts to be POSIX Compliant #1725
• @contractorwolf made their first contribution in rebase against your -np changes (thx) and add better python file to be used on the command line or as library #1744
• @Rakksor made their first contribution in talk-llama: optional wake-up command and audio confirmation #1765
• @rmmh made their first contribution in examples/server: implement "verbose_json" format with token details #1781
• @luciferous made their first contribution in whisper.android: Return output from benchmarks #1785
• @trixirt made their first contribution in libwhisper.so should be position independent #1792
• @mrienstra made their first contribution in Docs: try to make model options / model install methods clearer #1806
• @JacobLinCool made their first contribution in Add fields to verbose_json response and show examples on the home page #1802
• @valenting made their first contribution in server: Allow CORS request with authorization headers #1850
• @dscripka made their first contribution in added audio_ctx argument to main and server examples #1857
• @jumpers775 made their first contribution in fix openvino setup docs #1874
• @Theldus made their first contribution in main : check if input files exist before proceeding #1872
• @lbluep made their first contribution in Linking issue fix via Makefile when CUBLAS enabled in the WSL #1876 #1878
• @st-gr made their first contribution in openvino : fix convert-whisper-to-openvino.py for v2023.0.0 (#1870) #1890
• @abhilash1910 made their first contribution in Add SYCL logic in whisper #1863
• @Andrews54757 made their first contribution in Stream.wasm: Fix invalid memory access when no segments are returned #1902
• @gavin1818 made their first contribution in Update README to Recommend MacOS Sonoma for Core ML to avoid hallucination #1917
• @kenneth-ge made their first contribution in Add library versioning #1352
• @zhouwg made their first contribution in Fix SF(segment fault) issue in Android JNI #1929
• @F1L1Pv2 made their first contribution in Auto lowercase language parameter #1928
• @josharian made their first contribution in whisper : make beam candidate sort more stable #1943
• @joliss made their first contribution in Rename --audio-context to --audio-ctx, as per help text #1953
• @Man2Dev made their first contribution in Fedora dependencies needed (SDL2) #1970
• @tiagofassoni made their first contribution in libcuda.so.1 in PATH in Docker Container #1966
• @sanchit-gandhi made their first contribution in Improve support for distil-large-v3 #1982
• @ulatekh made their first contribution in Implemented command-style grammar in the main example. #1998
• @primenko-v made their first contribution in Support CUDA versions < 11.1 #2020
• @slashlib made their first contribution in Support for CPU BLAS build via Intel MKL #2024
• @rotemdan made their first contribution in Set stdin to binary mode on Windows. Fixes #2023 #2025
• @bradmurray-dt made their first contribution in Fix DTW memory access #2012
• @sixcircuit made their first contribution in fix missing reference to "model" variable in actual shell command run in whisper.nvim #2049
• @pprobst made their first contribution in feature/no timestamps node #2048

Full Changelog: v1.5.4...v1.5.5

---

This discussion was created from the release v1.5.5.

[timwu-ipevo]

Will executable files built for various accelerations also be released like past releases?

[ggerganov]

There was a last-minute regression in 1.5.5 (#2066) causing extra CPU usage in the main example. After fixing it #2070 I'm thinking of making a new minor release and will add the binaries there. Will give this a day or two before 1.5.6 to see if there are any other problems present

[SunnyOd]

Hi @ggerganov

I can see you've been adding binaries to the releases. Looks like they're only for windows though? Any chance of adding Linux binaries too? CUDA toolkit is a nightmare to install on so many distro's. I'm running Fedora so I am forced to use the CPU binaries as I can compile those okay

Btw thanks for this project it saves me hours every day, appreciate it!

[ggerganov]

AFAIAW creating binaries for anything else than Windows is quite tricky, so I don't think we will add this soon. Though, if there are contributions to make this work in a reasonable way, we can do it

[ulatekh]

@SunnyOd : I build whisper.cpp with CUDA under Fedora just fine. After installing the relevant CUDA repo, you'll need to build and install GCC 12.

[uckthis]

We use it in Subtitle Edit - it's the most efficient of all the engines available. Waiting for new EXE, though.

Thank you.

[ggerganov]

I'll focus the next day or two on whisper.cpp issues and try to push a v1.5.6 release with Win binaries

[ggerganov]

Plan is to release v1.6.0 with Flash Attention support today/tomorrow. Feedback on the new FA functionality is appreciated: #2152