The resulting hybrid models are higher-quality, more stable in training, and exhibit lower latency. Since the RNN decoder is much faster at inference time, we applied a variety of optimizations before coupling it with the transformer encoder. Transformer models have been demonstrated to be generally more effective at machine translation than RNN models, but our work suggested that most of these quality gains were from the transformer encoder, and that the transformer decoder was not significantly better than the RNN decoder. Following our work decoupling different aspects of model performance, we have replaced the original GNMT system, instead training models with a transformer encoder and an RNN decoder, implemented in Lingvo (a TensorFlow framework). Hybrid Model Architecture: Four years ago we introduced the RNN-based GNMT model, which yielded large quality improvements and enabled Translate to cover many more languages.

The improvements since the implementation of the new techniques over the last year are highlighted at the end of the animation.Īdvances for Both High- and Low-Resource Languages The quality improvements, which averaged +5 BLEU score over all 100+ languages, are visualized below.īLEU score of Google Translate models since shortly after its inception in 2006. These techniques span improvements to model architecture and training, improved treatment of noise in datasets, increased multilingual transfer learning through M4 modeling, and use of monolingual data. In this post, we share some recent progress we have made in translation quality for supported languages, especially for those that are low-resource, by synthesizing and expanding a variety of recent advances, and demonstrate how they can be applied at scale to noisy, web-mined data. Many techniques have demonstrated significant gains for low-resource languages in controlled research settings (e.g., the WMT Evaluation Campaign), however these results on smaller, publicly available datasets may not easily transition to large, web-crawled datasets. And while the research community has developed techniques that are successful for high-resource languages like Spanish and German, for which there exist copious amounts of training data, performance on low-resource languages, like Yoruba or Malayalam, still leaves much to be desired. Nevertheless, state-of-the-art systems lag significantly behind human performance in all but the most specific translation tasks. Posted by Isaac Caswell and Bowen Liang, Software Engineers, Google ResearchĪdvances in machine learning (ML) have driven improvements to automated translation, including the GNMT neural translation model introduced in Translate in 2016, that have enabled great improvements to the quality of translation for over 100 languages.

How accurate is google translate