Multilingual AI evaluation

Rating work in the format your pipeline expects, plus the notes that make the data useful:

• Response rating on your rubric, covering accuracy, helpfulness, fluency, tone, instruction following and safety, with written rationales wherever the rubric asks for them.
• Side-by-side preference judgments for comparing models, prompts or system instructions.
• Red-teaming: adversarial prompts written natively in the target language to surface harmful, biased or simply wrong output, including failures that never appear in English.
• Checks for locale-specific problems: formality and register, names, dates and addresses, idioms, and cultural references that do not carry over.
• Guideline feedback: where the rubric is ambiguous in a given language, with examples, so your guidelines improve along with the data.
• Work done in your platform or tooling, or in spreadsheets if that is what your team uses.

Every batch also comes with a note on what we saw beyond the individual scores: recurring failure types, the prompts that exposed them, and cases where the model was right but the rubric would mark it wrong. That note is usually where guideline improvements come from.

1. Guidelines. You share the rubric, examples and the tool. We read them, ask questions, and rate a calibration set.
2. Calibration. We compare our ratings with your gold answers or expectations and settle disagreements before volume work starts.
3. Rating. We work through the queue. Partner linguists handle other languages under the same guidelines, calibrated the same way. Items nobody can rate reliably are flagged rather than guessed.
4. Reporting. You receive the ratings, the rationales and a short note on patterns and guideline issues worth fixing.
5. 30-day fix window. For 30 days after delivery, if your QA flags ratings that do not follow the agreed guidelines, we redo them at no charge.

For red-teaming, we agree the risk categories and boundaries with you first, then log every successful attack with the exact prompt, the output and the language-specific reason it worked.

Confidentiality is standard for this work. We sign your NDA, work inside your tools where possible, and never reuse your prompts, outputs or guidelines elsewhere. Partner linguists sign a confidentiality agreement at least as strict before they see anything, as the confidentiality policy sets out.

Rating, evaluation, red-teaming

$35 / hour

about €32.20

Rating, evaluation and red-teaming are billed by the hour at the rate above, whether the task is quick preference judgments or slow adversarial work. We track time per task, so you can see exactly where the hours go and compare them with your own estimates. The calibration set also gives us both a realistic time per item, so the estimate for the full queue rests on evidence rather than guesswork. Invoices break the hours down by language and task type.

What moves the total: the number of languages, the length and difficulty of each item, how much written rationale the rubric requires, and how much calibration a new guideline needs before ratings settle. A clear rubric with good examples is the single biggest saving, because calibration is where ambiguous guidelines cost time.

Prices are in US dollars; the euro figures are approximate conversions for reference. For translation and linguistic QA of your own product content, rather than model output, see localization and LQA.