How to evaluate Arabic quality in an AI system before you buy it

A pattern repeats across the Gulf: a vendor demonstrates an AI system in English, it performs impressively, the contract is signed — and three months later the Arabic experience is quietly embarrassing. Citizens switch to the English interface. Staff stop using the assistant. The programme's credibility takes damage no slide deck repairs.

None of this is surprising once you know how these systems are built. Most were trained predominantly on English, evaluated on English benchmarks, and wrapped in safety filters tuned on English. Arabic capability varies wildly between models — and between versions of the same model. The only reliable answer is to measure it yourself, on your own workload, before you commit.

Why Arabic is genuinely harder to get right

Arabic presents specific engineering challenges that generic systems mishandle:

• Diglossia. Formal Modern Standard Arabic and spoken Gulf dialects differ substantially. A system trained mostly on news text reads formal documents adequately and stumbles on a WhatsApp-style citizen message.
• Code-switching. Real Gulf communication mixes Arabic and English mid- sentence — often with product names, dates and technical terms in Latin script inside Arabic text. Extraction and understanding routinely break here.
• Orthographic variation. The same word appears with or without hamza, with different taa marbuta handling, with or without diacritics. Names transliterate a dozen ways. Naive matching silently fails.
• Official terminology. Government, banking and clinical vocabulary has authorised Arabic forms. A model that improvises translations produces text your legal and communications teams cannot publish.
• Dual calendars and numerals. Hijri dates, Gregorian dates, Eastern and Western numerals — all coexist, often in one document.

Build the evaluation set from your own traffic

Public Arabic benchmarks are better than nothing, but they measure the wrong thing: performance on someone else's distribution. The evaluation that predicts your outcome is built from your material:

1. Sample 150–300 real items from the workload you are buying for — citizen queries, contracts, claims, emails — covering both languages and the mixtures between them.
2. Include the ugly cases deliberately: dialect, typos, scanned documents, mixed script, long documents. Production traffic is ugly; your evaluation should be too.
3. For each item, write down what a correct output looks like before testing. Post-hoc judgement drifts toward generosity.
4. Have bilingual reviewers score outputs on a simple scale — correct, usable with edits, wrong, harmful — and track scores separately for Standard Arabic, dialect, and mixed text.

Two hundred well-chosen items scored honestly will tell you more than any leaderboard.

Score the system, not the model

A common procurement error is evaluating the base model when what you are buying is a system: model plus retrieval plus prompts plus filters plus post-processing. Each layer can rescue or ruin Arabic quality. Retrieval tuned only for English embeddings will fetch the wrong documents no matter how good the model is; a safety filter tuned on English will block legitimate Arabic queries at surprising rates — a failure mode we cover in our security work as well, because it cuts both ways.

Test end to end, through the interface your users will actually use, in both languages, with the same rigour.

Make the numbers contractual

The evaluation is not just a selection tool — it is leverage. Write the scores into the contract: minimum acceptable accuracy on your evaluation set at acceptance, and the right to rerun the suite after every major update with remedies if quality regresses. Vendors who are confident in their Arabic accept this readily. Vendors who resist are telling you something useful.

Arabic-first is not a slogan for this region; it is table stakes for any system that faces citizens or staff. The organisations that measure before they buy end up with AI people actually use — in both languages. The ones that trusted the English demo end up writing the case study nobody wants to star in.