Self-Guided Contrastive Learning for BERT Sentence Representations

Information

Title Self-Guided Contrastive Learning for BERT Sentence Representations

Authors

Taeuk Kim, Kang Min Yoo, Sang-goo Lee

Year 2021 / 08

Keywords natural language processing, sentence representation, contrastive learning, pre-trained language models

Publication Type International Conference

Publication The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021)

Link url

Abstract

Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence rep- resentation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.