Arabic and the frontier

Leading models are usually measured in English. Arabic is spoken by hundreds of millions of people across many dialects, and the gap between Modern Standard Arabic and the Arabic people actually write in messages, support chats and social media is wide.

The question is plain: how well do the leading models handle Arabic as it is really used, in Modern Standard Arabic and in the major dialects, on tasks a business depends on?

• Tasks people rely on: understanding a request, answering from a document, summarising, writing a reply, and simple reasoning, each in Modern Standard Arabic and in dialects.
• Original material: prompts and reference answers written for this study, so the results do not depend on test sets the models may already have seen.
• Human grading: answers graded by native speakers of each dialect against a written rubric, with a share graded twice to measure agreement.
• Same conditions for every model: the same prompts, settings and dates, recorded with each result.
• Rerunnable: the prompts, the rubric and the grading code published so anyone can rerun the study on a newer model.

The list of dialects, the models and the number of prompts will be fixed and stated before the first run.

• The paper, with a Method section written so that a stranger can repeat it.
• The prompts, reference answers and rubric, and every graded answer.
• Dated versions; errata listed on the paper.
• One reader from outside the lab, named on the paper.