handy models

speech-to-text benchmarks for transcribe.cpp, the engine behind Handy

model detail

Where this model stands against the best model in each language at Q8_0, its speed on an AMD Ryzen 4750U laptop, and what the card says. Add a second model to compare them head to head.

model granite-speech-4.1-2b
AMD 4750U GPU4.6×
AMD 4750U CPU2.8×
Q8_0 size2.38 GB
frontier · AMD 4750U GPU0 of 6

verdict: Never first, and outside the leader's 95% CI in all 6 languages. Weakest relative to the field in Japanese (CER) (2.7× the leader's error), Portuguese (2.6×), and Spanish (2.1×). Faster than realtime on the AMD 4750U GPU (4.6×) and AMD 4750U CPU (2.8×).

license: Apache-2.0 · card languages: 6 · translate

standing by language · 6 benchmarked

Each bar compares this model with the best other model in that language, on a log scale: left of center it is ahead, right of center behind, and the other model is named on the row. The two error rates use the legend colors: this model first, the other model second. Faded bars are ties, where the 95% confidence intervals overlap. Sorted strongest standing first. Hover a row for full details.

granite-speech-4.1-2b error best other model error tie: the 95% CIs overlap

quantization · 6 usable languages at Q5_K_M

Q5_K_M is within noise on every one of 6 usable languages: 71% of the Q8_0 file for no measurable accuracy cost.

Q5_K_M file71% of Q8_01.70 GB against 2.38 GB
median error change+0.6%over 6 usable languages
measurably worse0 of 6above the Q8_0 95% confidence interval
worst usable languageJapanese (CER)6.30 → 6.59 (+0.29 pts)
measurably worse than Q8_0: above its 95% confidence interval within noise typical noise of one measurement
per-language numbers (6)
Japanese (CER) 6.305.75–6.88 6.59+0.29 +4.6%
French 7.616.90–8.36 7.70+0.09 +1.2%
Spanish 5.484.99–5.97 5.52+0.04 +0.7%
English 4.143.68–4.62 4.16+0.02 +0.5%
German 6.255.66–6.88 6.12−0.13 −2.1%
Portuguese 9.809.08–10.66 9.54−0.26 −2.7%

speed · all 73 models on the AMD 4750U GPU

× realtime, log scale. Left of the realtime marker, transcribing takes longer than the recording. Hover a dot for the model.

compare with

granite-4.0-1b-speechsame familygranite-speech-4.1-2b-narsame familygranite-speech-4.1-2b-plussame familygranite-speech-5.0-470m-turboctcsame familywhisper-large-v3ahead of it in 6 of 6 shared languagescanary-180m-flashclosest coverage, 4 languages in common