handy models

speech-to-text benchmarks for transcribe.cpp, the engine behind Handy

model detail

Where this model stands against the best model in each language at Q8_0, its speed on an AMD Ryzen 4750U laptop, and what the card says. Add a second model to compare them head to head.

model canary-1b-v2
AMD 4750U GPU14.1×
AMD 4750U CPU8.3×
Q8_0 size1.07 GB
frontier · AMD 4750U GPU16 of 25

verdict: Best in 11 of 25 languages, and tied with the leader in 4 more (overlapping 95% CIs). Weakest relative to the field in Greek (2.9× the leader's error), Ukrainian (1.9×), and Russian (1.6×). Faster than realtime on the AMD 4750U GPU (14.1×) and AMD 4750U CPU (8.3×).

license: CC-BY-4.0 · card languages: 25 · translate

standing by language · 25 benchmarked

Each bar compares this model with the best other model in that language, on a log scale: left of center it is ahead, right of center behind, and the other model is named on the row. The two error rates use the legend colors: this model first, the other model second. Faded bars are ties, where the 95% confidence intervals overlap. Sorted strongest standing first. Hover a row for full details.

canary-1b-v2 error best other model error tie: the 95% CIs overlap

quantization · 25 usable languages at Q5_K_M

Q5_K_M is within noise on every one of 25 usable languages: 73% of the Q8_0 file for no measurable accuracy cost.

Q5_K_M file73% of Q8_00.78 GB against 1.07 GB
median error change+0.9%over 25 usable languages
measurably worse0 of 25above the Q8_0 95% confidence interval
worst usable languageUkrainian10.58 → 10.98 (+0.40 pts)
measurably worse than Q8_0: above its 95% confidence interval within noise typical noise of one measurement
per-language numbers (25)
Ukrainian 10.589.87–11.27 10.98+0.40 +3.8%
Croatian 8.407.81–9.01 8.68+0.28 +3.3%
Hungarian 13.0612.39–13.80 13.45+0.39 +3.0%
Czech 8.567.91–9.23 8.80+0.24 +2.8%
Danish 11.3410.73–11.93 11.56+0.22 +1.9%
Dutch 6.285.63–6.98 6.40+0.12 +1.9%
German 4.464.05–4.89 4.53+0.07 +1.6%
Slovenian 12.7611.35–14.61 12.93+0.17 +1.3%
Russian 7.837.24–8.48 7.93+0.10 +1.3%
Estonian 12.7212.04–13.43 12.88+0.16 +1.3%
Latvian 10.419.79–11.09 10.54+0.13 +1.2%
Polish 6.886.32–7.44 6.96+0.08 +1.2%
English 4.474.03–4.95 4.51+0.04 +0.9%
Slovak 6.846.14–7.54 6.88+0.04 +0.6%
Finnish 8.868.21–9.55 8.91+0.05 +0.6%
Lithuanian 13.4512.74–14.16 13.52+0.07 +0.5%
Portuguese 4.504.04–5.01 4.51+0.01 +0.2%
Spanish 3.102.78–3.44 3.10+0.00 +0.0%
Greek 26.0225.01–26.98 25.99−0.03 −0.1%
Romanian 6.876.35–7.38 6.84−0.03 −0.4%
Bulgarian 9.228.43–10.04 9.10−0.12 −1.3%
Swedish 9.749.06–10.44 9.52−0.22 −2.3%
Maltese 19.7517.99–22.38 18.97−0.78 −3.9%
Italian 3.102.77–3.46 2.97−0.13 −4.2%
French 5.094.61–5.59 4.82−0.27 −5.3%

speed · all 73 models on the AMD 4750U GPU

× realtime, log scale. Left of the realtime marker, transcribing takes longer than the recording. Hover a dot for the model.

compare with

canary-180m-flashsame familycanary-1bsame familycanary-1b-flashsame familycanary-qwen-2.5bsame familywhisper-large-v3ahead of it in 12 of 25 shared languagesorukeetclosest coverage, 25 languages in common