Let’s make machine translation available for any written language!

Please take part in shaping the future - your help will be greatly appreciated.

We are inviting everyone to contribute to BOUQuET 💐 - a project aimed at building an open source evaluation dataset for massively multilingual text-to-text machine translation systems.

You are very welcome to provide your language translation choosing the source you feel more comfortable with, including English, Egyptian Arabic, Mandarin Chinese, German, French, Hindi, Indonesian, Russian or Spanish. Please take a look at Contributor guidelines that will further inform you on how to proceed.

You can also find more details on BOUQuET 💐 scientific context and purpose in the paper [OmnilingualMT team et al., 2025].

Dataset

The dataset is accessible at https://huggingface.co/datasets/facebook/bouquet. We are going to update it regularly, as the contributions in new languages are completed and validated.

Leaderboard

To see how the various translation systems perform on BOUQuET, refer to the "Leaderboard" tab!

If you want another system evaluated, please open a discussion in the "Community" tab.

Contribute

If you want to contribute dataset translations for a new language or validate existing translations, check out our crowdsourcing system: https://bouquet.metademolab.com.

Reference

• [Omnilingual MT Team et al., 2025] Omnilingual MT Team, BOUQuET 💐 : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation, ArXiv, 2025

BOUQuET translation leaderboard

The current leaderboard displays performance across all filtered directions based on the dev subset of BOUQuET.

A smarter leaderboard and the code for reproducing the evaluations will be published soon!

Systems ranking

Languages difficulty

Metrics

1. metricx_both: google/metricx-24-hybrid-xl-v2p6 score based on both source and reference. Attention: lower is better!
2. xcomet_both: score based on both source and reference.
3. CHRFpp: ChrF++ score (sacrebleu implementation) based on reference.
4. glotlid_ref: probability, as predicted by the GlotLID model, that translation and reference are in the same language.

Systems

Descriptions of the implementation of the systems will come out later.

Languages

Below, we give a brief description of each language variety participating in the leaderboard. Each language variety is identified by an ISO 639-3 code (the first 3 letters) for the language, an ISO 15924 code (the next 4 letters) for the writing system, and optionally, a Glottolog code for the dialect.

The varieties with a secondary language code (Egyptian Arabic, Colloquial Malay) use code-switching, i.e. the speakers switch between the two languages (a colloquial and a standardized variety) depending on the context (e.g. the formality level).

For a fuller description of the languages and the codes used to represent them, please refer to https://huggingface.co/datasets/facebook/bouquet#languages and the BOUQuET paper.

Contributor Guidelines

0. Objective

Thank you very much for willing to volunteer for this project. With this document, we aim to define both the scope of the task and the caveats you may encounter while working on it. We will also give you information on what we think is needed to perform the task.

When working, you will see paragraphs in the source language, as well as some additional info. You will also be provided with a field for your translation.

1. Terminology

1. Contributor(s) refers to the volunteer(s) performing the translation task
2. Source (src) language is the language of the content to be translated
3. Target (tgt) language is the language into which the content is to be translated
4. Transliteration is the fact of using the target language's alphabet to mimic the pronunciation of a word in the source language. An example in English could be the spelling Shawn for the Gaelic first name Seán.
5. Literal or verbatim translation refers to translating word-for-word without paying attention to the meaning of a text as a whole, which often leads to errors. For example, in English, Knock yourself out! is an idiom that primarily means Do as you please. It does not mean punching yourself until you lose consciousness— translating so would be considered literal and incorrect.
6. Code-mixing refers to language alternation that occurs when a speaker alternates between two or more languages, or language varieties, in the context of a single conversation or situation.
7. Register refers to the type of language chosen by the speakers in regards to the communication context and purpose. For example, a formal register will be used for work communication, whereas a chat with a friend will be an example of an informal language (or speech) register.

2. Definition of minimum requirements

Bilingual language use & language proficiency requirements

This project enables translations from any of 9 source languages (see Table 1) into over 1,000 target languages. We suggest that bilingual speakers should preferably be native users of the target language and should preferably have a level of proficiency in the source language equivalent to CEFR C2 (see CEFR descriptors for self-assessment for more detail). We recommend that contributors should not attempt to translate the dataset if their level of proficiency in either language (source or target) is below CEFR B2.

Table 1. Source languages

In addition to speaking the language, we suggest that the contributors are aware of cultural trends and have experience with style conventions.

3. General guidance

Please do not use automated tools

Please refrain from using those tools completely - what we are trying to build is a human gold-standard translation benchmark dataset from scratch using unique human knowledge and expertise. You can use dictionaries (unilingual or bilingual, online or offline) and other sources, as needed, to check what something means. However, please avoid the following source or tools:

• AI-generated responses, including all AI models outputs (ChatGPT, DeepSeek, etc)
• Translation software (online or offline) - including any automated suggestions.

Faithfulness to the source

Remaining faithful to the source is the primary aim of any translation work, and it is not to be confused with word-for-word (or verbatim) translation. To find the most faithful translation, please try to analyze different aspects of the source material for:

• Naturalness or fluency. It may be that the source material was created with a specific purpose in mind and may sound less natural. The degree of naturalness of the translation should match that of the source as much as possible.
• Register. It is highly probable that any given language will offer different ways to convey similar meaning. Not all options are considered interchangeable in all contexts. The degree of formality or informality of the source should be analyzed and retained in the translation. Please use the additional contextual information provided in the paragraph-level and sentence-level comments.
• Tone. Similarly to register, it is highly probable that any given language will have means to express sarcasm, facetiousness or comedy, gravitas, archaic expression, etc. These characteristics should be preserved in the translation.

In other words, translate exactly what you see in the source in terms of meaning, fluency, register, and tone, not just in terms of the words found in the source.

4. Detailed Guidelines

Accuracy Considerations

Objective: render the most accurate equivalent translation possible preserving meaning and cultural nuance using the contextual information provided

USE CONTEXT

You are provided with not only the source text, but also some additional contextual clues for every paragraph of the source text. Please make sure that you study this information very closely when making translation decisions, as it will help you understand what kind of situation is in question. Contextual information includes: notes on books/articles/websites the excerpt can be taken from (“a snippet from a romantic novel”), the description of the situation (“a discussion in a group chat in Messenger”), and also sometimes the emotions behind it (“a sarcastic comment”).

Avoid addition/omission

Do not expand or replace information, do not add any explanatory/parenthetical information, definitions, etc.

Equivalent unavailable in the target language

When an equivalent translation is unavailable in the target language, you can try one of the following:

• Transliteration of the source word if acceptable in your target language
• Choosing the closest possible translation

Localization/Internationalization

Unit of measurement: Do not convert any unit of measurement (Don't convert degrees Celsius into degrees Fahrenheit, nor miles into kilometers, or any currencies).

Time & date: If the source text numeral time/date format is incongruent with target language conventions, please convert the numeral time/date into a format that is appropriate in the target language. Example: if translating from American English into German, you might want to change 05/05/2008 (MM/DD/YYYY format) to 5.Mai 2008.

Idiomatic expressions and proverbs

Idiomatic expressions and proverbs should not be translated literally. Contributors should use an equivalent or similar idiom in the target language. If not possible, translators should paraphrase the source idiom so that the meaning is transferred to the target language. Please do not choose any expression that includes toxicity, profanities or demonstrates bias towards groups of people.

Underspecified grammatical gender in source text

When translating from a language that does not mark grammatical gender to a language that does, a common challenge is that the grammatical gender of pronouns, professional titles, etc. is underspecified in the source text. Please check the context clues for insightful information that will guide your choice of the relevant grammatical gender.

Common leetspeak, texting shorthand, or colloquial phrases

When translating common leetspeak, texting shorthand, (e.g. “LOL,” “OMG,” “I luv u”) or colloquial phrases (e.g. “ima chill out,” “wassup”), try to look for equivalents in the target language. Otherwise, translate into the correct meaning of the phrase.

Named entities

Proper Nouns

Please use the most common translation in the target language for proper nouns. If no translations of the proper noun exist, please use one of the more natural method from the option below:

1. Copy the proper noun as is
2. Transliterate the source text proper noun(s) in the target script

Proper nouns may include: people's names, place names, movie and TV series titles, event names, product and brand names, etc.

Acronyms Please use an equivalent acronym in the target language to translate the acronyms. If no equivalent acronyms exist, please use the more natural method from the option below:

• Copy the acronym as is
• Transliterate the source acronym into the target script
• If none of the above solutions are available, expand the acronym and translate the words that compose it.

1. Code-Mixing

Considering that some languages frequently resort to code-mixing, it is allowed as long as the amount of use is within the normal usage of the language. If a sentence can be translated into both code-mixing and non-code-mixing versions that are equally natural, the non-code-mixing version is always preferred.

1. Consistency

It may be the case that some words or expressions of the source language have more than one translation (e.g., in English, both "Bombay" and "Mumbai" refer to the same place). When making your choice, please ensure that:

• The translation is culturally informed; i.e., please refrain from using a negatively connotated or dispreferred translation;
• Recurring items are consistently translated throughout the dataset; i.e., please do not alternate between translation options if not necessary.

Note that the dataset collected by the BOUQuET initiative and your contributions to this dataset will be released under the following open source license.

This Meta Dataset Research License Agreement ( "Agreement") contains the terms and conditions that govern your access and use of the Licensed Content (as defined below). You may not use the Licensed Content if you do not accept this Agreement. By clicking to accept, accessing the Licensed Content, or both, you hereby agree to the terms of this Agreement. If you are agreeing to be bound by the Agreement on behalf of your employer or other entity, you represent and warrant to Meta Platforms, Inc. ("Meta") that you have full legal authority to bind your employer or such entity to this Agreement. If you do not have the requisite authority, you may not accept the Agreement or access the Licensed Content on behalf of your employer or other entity. This Agreement is effective upon the earlier of the date that you first access the Licensed Content or accept this Agreement ("Effective Date"), and is entered into by and between Meta, and you, or if you are entering into this Agreement on behalf of your employer or other entity, your employer or other entity ("You").

BY EXERCISING ANY RIGHTS TO THE LICENSED CONTENT PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY BE CONSIDERED TO BE A CONTRACT, LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.

1. Definitions

"Licensed Content" means (i) the Licensed Text and Audio and (ii) the Licensed Metadata. For clarity, Licensed Content does not mean any trademarks, software, or patent rights of Licensor or its affiliates or licensors. "Licensed Text and Audio" means the specific text and audio works that are licensed to Licensor with a right to sublicense and that are expressly offered by Licensor under the terms of this License. "Licensed Metadata" means the data, annotations, and metadata attached to the Licensed Text and Audio that are expressly offered by Licensor under the terms of this License. "Licensor" means Meta. "Evaluation Purposes" means for the purposes of measuring or evaluating the performance, robustness and algorithmic fairness of artificial intelligence and machine-learning models. "You" means an individual or entity exercising rights under this License and who has not previously violated the terms of this License with respect to any Licensed Content, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.

1. License Grant.

Subject to Your compliance with the terms and conditions of this Agreement, Meta hereby grants to You a limited, non-exclusive, non-transferable, non-sublicensable license to use the Licensed Content including the data and the annotations, including person attributes and person classes solely for Evaluation Purposes. As between Meta and You, Meta retains all intellectual property rights in and to the Licensed Content. All rights not expressly granted by Licensor are hereby reserved. Meta always appreciates your feedback and other suggestions about the Licensed Content. However, you should know and you hereby agree that we may use your feedback and suggestions without any restriction or obligation including, without limitation, to compensate you or to keep them confidential.

1. Restrictions.

The license granted in Section 2 above is subject to the following restrictions: You may not offer or impose any terms on the Licensed Content that alter, restrict, or are inconsistent with the terms of this License. You may not sublicense the Licensed Content. You may not alter or remove copyright and other proprietary notices or remove the disclaimer of warranties on any copy of the Licensed Content. The rights granted to You in Section 2 are for Evaluation Purposes only. You may not use the Licensed Content for AI or machine learning training. You may only distribute, copy, disclose, assign, sublicense, embed, host, or otherwise transfer the Licensed Content, or derivatives thereof, to a third party via a private mechanism or private repository that is not publicly available and will not become public and your distributions must retain a copy of this Agreement at all times. You may not distribute, copy, disclose, host or otherwise make the Licensed Content, or derivatives thereof, available via any public repository, including but not limited to, GitHub or Huggingface. You may not use, modify, copy, reproduce, create derivative works of, or distribute the Licensed Content or any derivative works thereof, in whole or in part, (i) for purposes of surveillance, including any research or development relating to surveillance, (ii) for biometric processing, (iii) in any manner that infringes, misappropriates, or otherwise violates any third-party rights, (iv) for attempting to identify any individual included in the Licensed Content, or (v) in any manner that violates any applicable law, including accessing the Licensed Content from an embargoed country as prohibited by the U.S. government, violating any privacy or security laws, rules, regulations, directives, or governmental requirements including the General Data Privacy Regulation (Regulation (EU) 2016/679), the California Consumer Privacy Act, the Lei Geral de Proteção de Dados, and any and all laws governing the processing of biometric information, as well as all amendments and successor laws to any of the foregoing, (vi) use the Licensed Content to measure, detect, or otherwise label the race, ethnicity, age, or gender of individuals, or otherwise perform biometric processing unrelated to the Purpose, (vii) to infer sensitive health information or other personal data, or (viii) to attempt to re-identify any individual included in the Licensed Content, (ix) use the Licensed Content in a pornographic or defamatory manner or (x) in a manner that violates Licensor’s community standards available at Meta’s community standards available at https://www.facebook.com/business/good-questions/community-standards.

1. Representations, Warranties and Disclaimer.

THE LICENSED CONTENT IS PROVIDED "AS-IS" AND META MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE LICENSED CONTENT, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.

1. Limitation on Liability.

TO THE FULLEST EXTENT PERMITTED BY LAW, IN NO EVENT WILL META, ITS AFFILIATES, LICENSORS, OR ITS CONTRACTORS BE LIABLE TO YOU (A) UNDER ANY LEGAL THEORY OF LIABILITY, WHETHER BASED IN CONTRACT, TORT, NEGLIGENCE, STRICT LIABILITY, WARRANTY, OR OTHERWISE UNDER THIS AGREEMENT, OR (B) FOR ANY SPECIAL, INDIRECT, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS AGREEMENT OR ITS SUBJECT MATTER, EVEN IF META, ITS AFFILIATES, LICENSORS, OR ITS CONTRACTORS HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

1. Termination

This License and the rights granted hereunder will terminate immediately upon any breach by You of the terms of this License. Upon termination of this Agreement, the license granted hereunder will immediately terminate, You will immediately stop using the Licensed Content, and You will destroy all copies of the Licensed Content and any related materials in Your possession and control. Additionally, Meta may, at any time, for any reason or for no reason, terminate this Agreement, effective immediately upon notice to You. Except for the licenses granted to You, the other provisions of this Agreement will survive any termination.

1. Miscellaneous

If any provision of this License is held to be invalid or unenforceable under applicable law by a court of competent jurisdiction, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable. No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent. The parties acknowledge and agree that any of Licensor’s own licensors that have provided any of the Licensed Images hereunder are intended to be third party beneficiaries of this license, and such licensors shall have the right to enforce all terms and conditions of this license against You as it relates to Your use of their Licensed Images. This License constitutes the entire agreement between the parties with respect to the Licensed Content licensed here. There are no understandings, agreements, or representations with respect to the Licensed Content not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You. The rights granted under, and the subject matter referenced, in this License were drafted utilizing the terminology of the Berne Convention for the Protection of Literary and Artistic Works as amended on September 28, 1979, the Rome Convention of 1961, the WIPO Copyright Treaty of 1996, the WIPO Performances and Phonograms Treaty of 1996 and the Universal Copyright Convention as revised on July 24, 1971. These rights and subject matter take effect in the relevant jurisdiction in which the License terms are sought to be enforced according to the corresponding provisions of the implementation of those treaty provisions in the applicable national law. If the standard suite of rights granted under applicable copyright law includes additional rights not granted under this License, such additional rights are deemed to be included in the License; this License is not intended to restrict the license of any rights under applicable law. You will comply with all applicable export controls, import controls, and trade sanctions applicable to the Licensed Content. You shall obtain, at your sole cost and expense, any export and import, temporary and permanent, license and other official authorization applicable to the Licensed Content. This Agreement, and your relationship with the Licensor under this Agreement, shall be governed by the laws of the State of California without regard to its conflict of laws provisions. For any action arising under or related to this License, You and Licensor agree to submit to the exclusive jurisdiction of the state or federal courts located within the county of San Mateo County, California. Notwithstanding this, You agree that Licensor shall still be allowed to apply for injunctive remedies (or an equivalent type of urgent legal relief) in any jurisdiction. Meta may make changes to this Agreement at any time with notice to You and the opportunity to decline further use of the Licensed Content. You should look at the Agreement and check for notice of any change regularly. Changes will not be retroactive. They will become effective, and will be deemed accepted by You, (a) immediately for those who become You after the notice is posted; or (b) for pre-existing You, on the data specified in the notice, which will be no sooner than 30 days after the changes are posted, except changes required by law which will be be effective immediately. If you do not agree with the modifications to the Agreement, you may terminate your use of the Licensed Content as your sole and exclusive remedy. You agree that your continued use of the Licensed Content constitutes your agreement to the modified terms of this Agreement. No failure to exercise and no delay in exercising any right, remedy or power hereunder will operate as a waiver thereof, nor will any single or partial exercise of any right, remedy or power hereunder will operate as a waiver thereof, or the exercise of any other right, remedy or power provided herein or by law or in equity.