Benchmarking Arabic OCR and translation on an 8 GB laptop GPU

A police officer gets handed a document in Arabic. No interpreter in the building. What happens next?

That's the question I spent ten weeks on for lingomatch, a Gov/Justice-Tech startup in Saarbrücken. Can software produce a usable German gist on modest hardware? And more importantly, can it know when its own output shouldn't be trusted?

One thing before we start: this is a triage tool, not certified translation. The output tells an officer whether to call a human interpreter. It never replaces one. That framing shaped every decision below.

The setup: a laptop RTX 4060 with 8 GB of VRAM, public data (KITAB-Bench, FLORES-200, Arabic Wikipedia) and open-weight models. All sixteen notebooks are public. Here are the four lessons I'd keep.

1. The licence killed my best model before the benchmark could

For Arabic→German the obvious pick was NLLB, Meta's 200-language model. It worked well. I was happy.

Then a teammate caught something no benchmark shows: NLLB's weights are CC-BY-NC. Non-commercial. For a company that intends to ship, that's dead on arrival. Doesn't matter how good the BLEU is.

Here's the thing: every model comparison should start with the question nobody puts on the leaderboard. What are you actually allowed to ship?

The Apache-2.0 candidate was Opus-MT-ar-de from Helsinki-NLP. Roughly 300 MB against NLLB-1.3B's multi-gigabyte footprint. On short sentences it matched NLLB (BLEU 43.8 against 42.4) at about a ninth of the memory, 2.4x the speed, running happily on CPU. Tempting conclusion: migration free of charge.

It doesn't hold. On long legal text Opus-MT fell apart in ways the short-sentence benchmark never hinted at:

• It truncated clauses.
• It collapsed a philosophical quotation into a repetition loop.
• It mangled proper nouns.

NLLB stayed structurally coherent the whole time. Same models, longer inputs, opposite ranking. So the recommendation became conditional: Opus-MT for short documents, sentence-level reassembly to keep inputs short, escalation when documents run long. A choice that looked settled after notebook 08 got properly humbled by notebook 09. Which is exactly why notebook 09 existed.

2. The best translator scored the worst BLEU

TranslateGemma 4B produced the best German of anything I tested. It fixed every error the smaller models made on the hard sentences, including a Bossuet quotation that both NLLB and Opus-MT butchered.

It also scored the lowest BLEU of the lot.

Why? BLEU counts n-gram overlap against a reference, so it punishes good paraphrase. If I'd trusted the metric alone, I'd have ranked the best model last. Read the outputs, not just the scoreboard.

The same model gave me the most instructive failure of the whole project. TranslateGemma accepts images directly, so I tried skipping OCR entirely: image in, German out. On an image it couldn't properly read, it didn't error out. It didn't produce garbage either.

It hallucinated a complete, fluent, plausible German document that had nothing to do with the input.

For a casual app, that's a quirk. For a tool whose output influences what a police officer does next, it's disqualifying. A cascade (OCR first, then text translation) is 35x cheaper and every stage of it can be inspected and flagged. Auditability beat quality. Sounds wrong for a leaderboard. It was right for this product.

3. Accuracy is table stakes. Knowing when you're wrong is the product.

Surya OCR on 50 KITAB-Bench documents, evaluated per-sample: 31 of 50 reached usable quality (character error rate under 10%). Failures clustered in degraded scans and stylized fonts.

Honestly? That number alone is almost useless for deployment. The real question is whether the system can tell its 19 failures apart from its 31 successes on its own.

Mostly, yes. Surya's per-line confidence genuinely predicts failure: a document-level confidence router flagged 4 of 4 catastrophic OCR failures in the evaluation set. That router is the heart of the design: a low-confidence document becomes an automatic "call an interpreter", never a wrong translation delivered confidently.

Confidence alone has blind spots, so the final design layers four signals:

• Confidence routing catches the failures the engine knows about.
• Perplexity scoring catches implausible text, after I fixed a confound where Arabic diacritics inflated perplexity on perfectly good text.
• Script-ratio and repetition heuristics catch contamination plus the hallucination loops generative OCR falls into. Perplexity misses those.
• Model disagreement: run PaddleOCR as an independent second engine and flag documents where the two diverge. Two different architectures rarely hallucinate the same way.

One sample in fifty evaded all four signals. Wrong but confident, plausible, clean, consistent across engines. I consider that the most valuable result in the project, because it's the empirical answer to "can we remove the human from the loop?"

No. And under the EU AI Act's human-oversight requirements for high-risk systems, that answer needed to be demonstrated, not assumed.

4. Field conditions are a preprocessing problem before they're a model problem

Real inputs aren't clean scans. They're phone photos: skewed, shadowed, creased.

Before reaching for a bigger OCR model, notebook 16 tried classical preprocessing: projection-profile deskew plus CLAHE contrast recovery. On a multi-line Arabic page degraded with skew and shadow, that recovered about a third of the lost OCR quality (CER from 7.6% back to 4.9%) for a few milliseconds of OpenCV.

The cheapest model upgrade is often not a model.

Starting the same project tomorrow? Here's what I'd tell you

• Check the licence before the benchmark. It can end the comparison early.
• Benchmark on inputs that look like production, especially the ugly ones. Short-input results don't generalize and you won't know until you test.
• Metrics are instruments, not verdicts. When BLEU and your eyes disagree, investigate before trusting either.
• Build the distrust machinery with the same seriousness as the pipeline. Confidence, perplexity, heuristics, disagreement: each catches something the others miss. Something still gets through.
• 8 GB of VRAM is enough to answer real feasibility questions, if you pick model sizes deliberately and quantise without shame.

The full notebooks are in the repo, including raw outputs of both OCR engines on all 50 samples so you can reproduce the disagreement analysis without a GPU.