DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa (Decoding-enhanced BERT with disentangled attention) is a new model architecture that improves BERT and RoBERTa using two novel techniques. The first is a disentangled attention mechanism in which each word is represented by two vectors encoding its content and position separately, and attention weights are computed with disentangled matrices over the words' contents and relative positions. The second is an enhanced mask decoder that replaces the output softmax layer to predict masked tokens during pretraining.

These two techniques significantly improve pretraining efficiency and downstream task performance. A DeBERTa model trained on only half the training data consistently outperformed RoBERTa-Large across a range of NLP tasks, improving MNLI by +0.9% (90.2% vs 91.1%), SQuAD v2.0 by +2.3% (88.4% vs 90.7%), and RACE by +3.6% (83.2% vs 86.8%). The gains with less data, plus the public release of code and pretrained models, made DeBERTa an influential advance in pretrained language modeling.