On The Use Of BERT For Neural Machine Translation · The Large Language Model Bible Contribute to LLM-Bible

On The Use Of BERT For Neural Machine Translation

Clinchant Stéphane, Jung Kweon Woo, Nikoulina Vassilina. Arxiv 2019

[Paper]    
Applications BERT Model Architecture Security Training Techniques

Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness.

Similar Work