pith. sign in

arxiv: 2106.16138 · v2 · pith:NON6BKWCnew · submitted 2021-06-30 · 💻 cs.CL

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi , Shaohan Huang , Li Dong , Shuming Ma , Bo Zheng , Saksham Singhal , Payal Bajaj , Xia Song
show 3 more authors
Xian-Ling Mao Heyan Huang Furu Wei
This is my paper
classification 💻 cs.CL
keywords cross-lingualmodelpre-trainingtasksxlm-edetectionlanguagemultilingual
0
0 comments X
read the original abstract

In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

    cs.CL 2021-11 accept novelty 6.0

    DeBERTaV3 improves DeBERTa by switching to replaced token detection pre-training and using gradient-disentangled embedding sharing, reaching 91.37% on GLUE and new SOTA on XNLI zero-shot.