Transcription WorkBench #6303

lutangar · 2024-03-29T09:36:07Z

Description

┬───────────────────────┬──────────────────────┬──────────┬────────┬───────────────────────┐
│          WER          │         CER          │ duration │ model  │        engine         │
┼───────────────────────┼──────────────────────┼──────────┼────────┼───────────────────────┤
│ '30.373831775700932%' │ '9.824046920821115%' │  '31s'   │ 'tiny' │   'openai-whisper'    │
│ '25.233644859813083%' │      undefined       │  '12s'   │ 'tiny' │ 'whisper-ctranslate2' │
│ '35.046728971962615%' │      undefined       │  '18s'   │ 'tiny' │ 'whisper-timestamped' │
┴───────────────────────┴──────────────────────┴──────────┴────────┴───────────────────────┘
┬───────────────────────┬──────────────────────┬──────────┬─────────┬───────────────────────┐
│          WER          │         CER          │ duration │  model  │        engine         │
┼───────────────────────┼──────────────────────┼──────────┼─────────┼───────────────────────┤
│ '15.887850467289718%' │      undefined       │ '2m 51s' │ 'small' │   'openai-whisper'    │
│ '16.355140186915886%' │ '18.10850439882698%' │  '42s'   │ 'small' │ 'whisper-ctranslate2' │
│ '16.822429906542055%' │      undefined       │ '1m 5s'  │ 'small' │ 'whisper-timestamped' │
┴───────────────────────┴──────────────────────┴──────────┴─────────┴───────────────────────┘

💻 i7-7500U
🎥 🇫🇷 1m28s
https://podeduc.apps.education.fr/numerique-educatif/video/21893-communiquer-lors-dune-classe-transplantee/

Related issues

Has this been tested?

👍 yes, I added tests to the test suite

see specific transcriptions tests https://github.com/larriereguichet/PeerTube/actions/workflows/transcription.yml

💭 no, because this PR is a draft and still needs work
🙅 no, because this PR does not update server code
🙋 no, because I need help

Todo

Screenshots

socket-security · 2024-04-16T15:50:33Z

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report↗︎

socket-security · 2024-04-29T15:10:25Z

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/linkify-it@5.0.0	None	`0`	55.4 kB	vitaly
npm/lodash-es@4.17.21	eval	`0`	636 kB	bnjmnt4n
npm/lodash@4.17.21	None	`0`	1.41 MB	bnjmnt4n
npm/logform@2.6.0	None	`+4`	307 kB	dabh
npm/lru-cache@10.2.0	None	`0`	458 kB	isaacs
npm/magnet-uri@7.0.5	None	`0`	14.5 kB	webtorrent-bot
npm/markdown-it-emoji@3.0.0	None	`0`	344 kB	vitaly
npm/markdown-it@14.0.0	None	`+1`	794 kB	vitaly
npm/maxmind@4.3.18	filesystem, network	`0`	22.2 kB	runk
npm/memoizee@0.4.15	None	`+3`	125 kB	medikoo
npm/merge2@1.4.1	None	`0`	8.9 kB	zensh
npm/methods@1.1.2	network	`0`	5.29 kB	dougwilson
npm/mime-types@2.1.35	None	`+1`	224 kB	dougwilson
npm/mime@1.6.0	environment, filesystem	`0`	51.7 kB	broofa
npm/minimist@1.2.8	None	`0`	54.5 kB	ljharb
npm/mocha@10.3.0	environment, eval, filesystem	`+10`	2.84 MB	joshuakgoldberg
npm/moo@0.5.2	None	`0`	32.7 kB	tjvr
npm/morgan@1.10.0	eval Transitive: unsafe	`+3`	57.2 kB	dougwilson
npm/ms@2.1.2	None	`0`	6.84 kB	styfle
npm/multer@1.4.5-lts.1	filesystem	`+10`	235 kB	linusu
npm/next-tick@1.1.0	None	`0`	7.65 kB	medikoo
npm/node-media-server@2.6.2	filesystem, network, shell Transitive: environment	`+1`	2.23 MB	illuspas
npm/nodemailer@6.9.9	environment, filesystem, network, shell	`0`	500 kB	andris
npm/normalize-path@3.0.0	None	`0`	9.22 kB	jonschlinkert
npm/object-assign@4.1.1	None	`0`	5.49 kB	sindresorhus
npm/obuf@1.1.2	None	`0`	19.1 kB	indutny
npm/omggif@1.0.10	None	`0`	38.5 kB	deanm
npm/once@1.4.0	None	`+1`	7.01 kB	isaacs
npm/opentelemetry-instrumentation-sequelize@0.40.0	None	`0`	124 kB	aspecto-release-bot
npm/otpauth@9.2.2	None	`0`	505 kB	hectormolinero
npm/p-queue@8.0.1	None	`+1`	109 kB	sindresorhus
npm/pako@1.0.11	None	`0`	788 kB	vitaly
npm/parse-torrent@11.0.16	filesystem, network	`0`	27.4 kB	webtorrent-bot
npm/password-generator@2.3.2	None	`0`	12.9 kB	bermi
npm/path-key@3.1.1	None	`0`	4.55 kB	sindresorhus
npm/pg-connection-string@2.6.2	filesystem	`0`	9.1 kB	brianc
npm/pg-numeric@1.0.2	None	`0`	4.17 kB	charmander
npm/pg@8.11.3	environment, network	`+3`	121 kB	brianc
npm/picomatch@2.3.1	None	`0`	90 kB	mrmlnc
npm/piscina@4.3.2	unsafe	`0`	295 kB	metcoder95
npm/pixelmatch@5.3.0	None	`+1`	656 kB	mourner
npm/pngjs@7.0.0	None	`0`	650 kB	lukeapage
npm/postgres-array@2.0.0	None	`0`	4.9 kB	bendrucker
npm/postgres-bytea@1.0.0	None	`0`	3.06 kB	bendrucker
npm/postgres-date@1.0.7	None	`0`	5.92 kB	bendrucker
npm/postgres-interval@1.2.0	None	`0`	6.73 kB	bendrucker
npm/postgres-range@1.1.4	None	`0`	10.1 kB	martianboy
npm/prompt@1.3.0	Transitive: environment	`+3`	1.05 MB	caub
npm/proxy-addr@2.0.7	None	`+2`	63.4 kB	dougwilson
npm/proxy@2.1.1	filesystem, network	`0`	51.5 kB	tootallnate
npm/pug@3.0.2	environment, eval, filesystem	`0`	59.7 kB	pug-bot
npm/punycode@2.3.1	None	`0`	33.5 kB	google-wombot
npm/queue-microtask@1.2.3	None	`0`	8.37 kB	feross
npm/queue-tick@1.0.1	None	`0`	3.03 kB	mafintosh
npm/randombytes@2.1.0	None	`0`	6.36 kB	cwmma
npm/range-parser@1.2.1	None	`0`	8.46 kB	dougwilson
npm/readable-stream@3.6.2	environment	`0`	124 kB	matteo.collina
npm/readdirp@3.6.0	filesystem	`0`	20.5 kB	paulmillr
npm/redis-errors@1.2.0	None	`0`	8.85 kB	bridgear
npm/reflect-metadata@0.2.1	None	`0`	241 kB	rbuckton
npm/resolve-dir@1.0.1	Transitive: environment, filesystem	`+7`	63.8 kB	phated
npm/resolve@1.22.8	environment, filesystem	`+2`	159 kB	ljharb
npm/run-parallel@1.2.0	None	`0`	6.56 kB	feross
npm/safe-buffer@5.1.2	None	`0`	31.7 kB	feross
npm/safer-buffer@2.1.2	None	`0`	42.3 kB	chalker
npm/sanitize-html@2.11.0	None	`+2`	196 kB	boutell
npm/semver@7.6.0	None	`+1`	109 kB	npm-cli-ops
npm/sequelize-typescript@2.1.6	None	`0`	216 kB	robinbuschmann
npm/sequelize@6.33.0	filesystem	`+5`	7.42 MB	sdepold
npm/short-uuid@4.2.2	None	`+2`	193 kB	oculus42
npm/sitemap@7.1.1	filesystem, shell	`+2`	1.87 MB	derduher
npm/socket.io-client@4.7.4	None	`+1`	1.37 MB	darrachequesne
npm/socket.io@4.7.4	filesystem, network	`+1`	1.38 MB	darrachequesne
npm/sql-formatter@15.2.0	None	`+1`	2.93 MB	nene
npm/srt-to-vtt@1.1.3	None	`+1`	24.3 kB	mafintosh
npm/streamx@2.16.1	None	`0`	49.3 kB	mafintosh
npm/strip-json-comments@3.1.1	None	`0`	6.96 kB	sindresorhus
npm/supertest@7.0.0	network	`0`	24 kB	titanism
npm/swagger-cli@4.0.4	None	`0`	10.9 kB	jamesmessinger
npm/through@2.3.8	None	`0`	12.5 kB	dominictarr
npm/tsc-watch@6.0.4	shell	`+9`	152 kB	gilamran
npm/tslib@2.6.2	None	`0`	84 kB	typescript-bot
npm/tsx@4.7.1	None	`0`	394 kB	hirokiosame
npm/tweetnacl@0.14.5	None	`0`	174 kB	dchest
npm/type-check@0.4.0	None	`+1`	57.9 kB	gkz
npm/type-is@1.6.18	None	`+1`	29.6 kB	dougwilson
npm/typescript@5.2.2	None	`0`	40.6 MB	typescript-bot
npm/unpipe@1.0.0	None	`0`	4.31 kB	dougwilson
npm/useragent@2.3.0	filesystem, unsafe	`+2`	274 kB	3rdeden
npm/util-deprecate@1.0.2	None	`0`	5.48 kB	tootallnate
npm/validator@13.11.0	None	`0`	751 kB	profnandaa
npm/vary@1.1.2	None	`0`	8.75 kB	dougwilson
npm/webfinger.js@2.7.1	network	`0`	46.4 kB	silverbucket
npm/webtorrent@2.1.36	filesystem, network	`+2`	5.78 MB	webtorrent-bot
npm/winston@3.11.0	filesystem, network Transitive: environment	`+10`	469 kB	dabh
npm/ws@8.16.0	environment, network	`0`	141 kB	lpinca
npm/xtend@4.0.2	None	`0`	6.46 kB	raynos
npm/yauzl@3.1.0	filesystem	`+1`	92.9 kB	thejoshwolfe

🚮 Removed packages: npm/@babel/code-frame@7.23.5, npm/@babel/parser@7.23.9, npm/@babel/types@7.23.9, npm/@jridgewell/trace-mapping@0.3.22, npm/@types/express-serve-static-core@4.17.43, npm/ansi-styles@4.3.0, npm/application-config@3.0.0, npm/bare-events@2.2.0, npm/bare-os@2.2.0, npm/bare-path@2.1.0, npm/bitfield@4.2.0, npm/boolean@3.2.0, npm/braces@3.0.2, npm/chalk@4.1.2, npm/chrome-dgram@3.0.6, npm/cli-table3@0.6.3, npm/cross-fetch-ponyfill@1.0.3, npm/duplexify@3.7.1, npm/error@7.0.2, npm/es-abstract@1.22.4, npm/es5-ext@0.10.62, npm/es6-symbol@3.1.3, npm/fast-glob@3.3.1, npm/fetch-blob@3.2.0, npm/get-tsconfig@4.7.2, npm/glob@8.1.0, npm/hasown@2.0.1, npm/html-to-text@9.0.5, npm/htmlparser2@8.0.2, npm/k-bucket@5.1.0, npm/netrc-parser@3.1.6

View full report↗︎

chore: fiddling around some more chore: add ctranslate2 and timestamped chore: add performance markers chore: refactor test chore: change worflow name chore: ensure Python3 chore(duration): convert to chai/mocha syntahx chore(transcription): add individual tests for others transcribers chore(transcription): implement formats test of all implementations Also compare result of other implementation to the reference implementation chore(transcription): add more test case with other language and models size and local model chore(test): wip ctranslate 2 adapat

chore(test): clean a bit chore(test): clean a bit chore(test): refacto timestamed spec chore(test): update workflow chore(test): fix glob expansion with sh chore(test): extract some hw info chore(test): fix async tests chore(benchmark): add model info

feat(transcription): extract run concept in own class an run more bench chore(transcription): somplify run object only a uuid is now needed and add more benchmark scenario

docs(transcription): add local model usage

docs(timestamped): clean

Introduce whisper builtin model.

Forbid transcript creation without a language. Add `languageDetection` flag to an engine and some assertions. Fix an issue in `whisper-ctranslate2` : Softcatala/whisper-ctranslate2#93

Update existing time function to output an integer number of seconds and add a ms human-readable time formatter with hints of tests.

Thanks to this recent fix in Jiwer <3 https://github.com/jitsi/jiwer/issues/873

I'm not very happy with the TranscriptFileEvaluator constructor... suggestions ? chore(JiWer): add usage in README docs(jiwer): update JiWer readme

chore(transcription): remove transcription workflow docs(transcription): add benchmark info

…ate2 transcriber chore(transcription): ctranslate2 assertion chore(transcription): ctranslate2 assertion

chore(transcription): add download and unzip utils functions chore(transcription): download & unzip models fixtures chore(transcription): zip

lutangar marked this pull request as draft March 29, 2024 09:37

lutangar changed the title ~~WIP~~ Transcription WorkBench Mar 29, 2024

lutangar force-pushed the transcription-backend-workbench branch from 9dd15b8 to 1d27252 Compare April 23, 2024 08:26

lutangar force-pushed the transcription-backend-workbench branch from 2535ac1 to 380bd49 Compare April 29, 2024 15:24

lutangar force-pushed the transcription-backend-workbench branch from 4745d30 to c9d2385 Compare May 31, 2024 14:39

lutangar added 22 commits May 31, 2024 16:41

feat(transcription): allow use of a local mode in timestamped-whisper

56a315c

feat(transcription): extract run and profiling info in own value object

abfe46e

feat(transcription): extract run concept in own class an run more bench chore(transcription): somplify run object only a uuid is now needed and add more benchmark scenario

docs(transcription): creates own package readme

de3a1c1

docs(transcription): add local model usage

chore(transcription): make openai comparison passed

6aefca2

docs(timestamped): clean

fix(transcription): use fr video for better comparision

21dfee5

chore(transcription): change transcribers transcribe method signature

6e578a8

Introduce whisper builtin model.

fix(transcription): activate language detection

a042df4

Forbid transcript creation without a language. Add `languageDetection` flag to an engine and some assertions. Fix an issue in `whisper-ctranslate2` : Softcatala/whisper-ctranslate2#93

chore(transcription): use PeerTube time helpers instead of custom ones

2162bf9

Update existing time function to output an integer number of seconds and add a ms human-readable time formatter with hints of tests.

chore(transcription): use PeerTube UUID helpers

d4a84d7

chore(transcription): enable CER evaluation

4807078

Thanks to this recent fix in Jiwer <3 https://github.com/jitsi/jiwer/issues/873

chore(jiwer): creates JiWer package

ff6e6e5

I'm not very happy with the TranscriptFileEvaluator constructor... suggestions ? chore(JiWer): add usage in README docs(jiwer): update JiWer readme

chore(transcription): use FunMOOC video in fixtures

a11c897

chore(transcription): add proper english video fixture

746978e

chore(transcription): use os tmp directory where relevant

d173ed6

chore(transcription): move benchmark out of tests

b49a85e

chore(transcription): remove transcription workflow docs(transcription): add benchmark info

fix(transcription): use ms precision in other transcribers

1ebf187

chore(transcription): simplify most of the tests

0ab4b7d

chore(transcription): remove slashes when building path with join

7c20b55

chore(transcription): make fromPath method async

003fa31

chore(transcription): assert path to model is a directory for CTransl…

9a75b42

…ate2 transcriber chore(transcription): ctranslate2 assertion chore(transcription): ctranslate2 assertion

chore(transcription): add preinstall script for Python dependencies

e279398

lutangar force-pushed the transcription-backend-workbench branch from c9d2385 to 855e672 Compare May 31, 2024 14:42

chore(transcription): add download and unzip utils functions

3bffa61

chore(transcription): add download and unzip utils functions chore(transcription): download & unzip models fixtures chore(transcription): zip

lutangar force-pushed the transcription-backend-workbench branch from 855e672 to 3bffa61 Compare May 31, 2024 14:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transcription WorkBench #6303

Transcription WorkBench #6303

lutangar commented Mar 29, 2024 •

edited

socket-security bot commented Apr 16, 2024 •

edited

socket-security bot commented Apr 29, 2024 •

edited

Transcription WorkBench #6303

Are you sure you want to change the base?

Transcription WorkBench #6303

Conversation

lutangar commented Mar 29, 2024 • edited

Description

Related issues

Has this been tested?

Todo

Screenshots

socket-security bot commented Apr 16, 2024 • edited

socket-security bot commented Apr 29, 2024 • edited

lutangar commented Mar 29, 2024 •

edited

socket-security bot commented Apr 16, 2024 •

edited

socket-security bot commented Apr 29, 2024 •

edited