BLEU Score is an evaluation metric to measure the quality of machine translation. It is widely used in natural language processing (NLP) to measure the quality of machine generated translations. It is an entire calculation metric, based on precision of the machine generated translation.
BLEU (bi-Lingual Evaluation Understudy) is one of the most widely accepted and used metrics for evaluating system performance in the field of machine translation. Machine translation (MT) is the process of automatically translating text from one human language into another. The BLEU score is a measure of how closely the output of a machine translation system matches human translations for the same text.
The BLEU score is calculated by comparing the machine generated translation to existing translation from an expert human translator. For each sentence in the test document (a so-called translation segment), precision is calculated by counting the number of words appearing in the machine translation that also are found to appear in the human translation. The score is calculated by taking the harmonic mean of the precision for every translation segment.
The BLEU score is not without some limitations. One limitation is that it only evaluates a model’s performance on a lexical-level. This means that even if the model produces translations with perfect lexical accuracy, it doesn’t imply semantic accuracy and overall fluency. A second limitation is the reliance on a single set of human-translations which would create bias when compared with different experts’ translations.
Unlike other metrics like BLEU ,i.e. Levenshtein distance, BLUE score does not consider the server architecture and interface. Because of this, it is considered suitable for comparing different language models without any bias brought about by server architecture or interface structure.
The BLEU score has since become standard baseline in the evaluation of machine translation systems and is widely used when new models of machine translation are developed. The success of BLEU in machine translation and its popularity in NLP has broadened its success to other tasks in Machine Learning, such as summarization, image captioning, and speech recognition.
