How to use the bleu score calculator
- Paste one or more reference translations into the first box, one per line. In one-sentence mode each line is another acceptable version of the same sentence.
- Paste the candidate translation into the second box.
- Choose the mode. Use many lines, paired when you have a list of sentences and want one score for the whole set, with line 1 matched to line 1 and so on.
- Pick a smoothing choice and whether to ignore capital letters, then read the score, the brevity penalty and the precision for each n-gram length.
Formula
BLEU = BP x exp( (ln p1 + ln p2 + ln p3 + ln p4) / 4 )
p(n) = clipped matching n-grams / n-grams in the candidate
BP = 1 if the candidate is not shorter, else exp(1 - reference length / candidate length)
Score shown from 0 to 100
Clipping means a candidate word cannot be counted more times than it appears in the reference. With several references the highest count from any one of them is used, and the reference length closest to the candidate length sets the brevity penalty.
A cat on a mat
The reference is "the cat is on the mat today" and the candidate is "the cat is on a mat today". Both have seven words, so the brevity penalty is 1.
Six of the seven single words match, 4 of 6 word pairs, 2 of 5 triples and 1 of 4 four-word runs. The score is the fourth root of 6/7 x 4/6 x 2/5 x 1/4, which is 0.4889, so BLEU is 48.892.
What BLEU measures
BLEU compares a candidate with reference texts by counting shared runs of words. Matching single words reflects word choice, and matching longer runs reflects word order and fluency. The brevity penalty stops a very short candidate from scoring well just because everything it says is in the reference.
The score is meant for a whole set of sentences. On one sentence it is noisy, and it is 0 whenever the candidate shares no run of four words with the reference, even if the translation is good. The add 1 smoothing option gives short sentences a small count for the longer runs so the score is not forced to 0.
Limits worth knowing
BLEU sees only words, not meaning. A good paraphrase can score low and a stiff word-for-word text can score high. Tokenization also matters. This page splits at spaces only, so a comma stuck to a word makes a different token from the word without it. Published BLEU scores often use a standard tokenizer, so scores here match them only when the text is already tokenized the same way.
Use BLEU to compare versions of one system on one test set, with several references if you can. Do not compare scores across test sets or languages.
Assumptions and limits
- Tokens are the pieces between spaces. No other splitting or normalizing is done except the optional lowercase.
- The maximum n-gram length is 4, which is the usual BLEU-4.
- A candidate with fewer than 4 words has no 4-gram precision, so unsmoothed BLEU is 0.
- Nothing is uploaded. All text stays in your browser.
Frequently asked questions
Why is my BLEU 0?
If no run of four candidate words matches a run in a reference, the unsmoothed score is 0. Try the add 1 smoothing option, or use more sentences in corpus mode.
What is the brevity penalty?
A factor below 1 applied when the candidate has fewer words than the closest reference. It is exp(1 - reference length / candidate length).
How do multiple references work?
Each n-gram in the candidate can match whichever reference contains it the most times, and the reference whose length is closest to the candidate sets the brevity penalty.
Does it match sacrebleu?
With no tokenizer in sacrebleu the scores match for the same texts. Standard sacrebleu tokenizes punctuation, so tokenize the text the same way first.