IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?
Information
Title
IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?
Authors
Jaeyoul Shin, Taeuk Kim, Sang-goo Lee
Year
2020 / 7
Keywords
natural language processing, pre-trained language model, emphasizing
Acknowledgement
HPC
Publication Type
International Workshop
Publication
International Workshop on Semantic Evaluation (SemEval 2020)
Link
url
Abstract
We propose a novel method that enables us to determine words that deserve to be emphasized from written text in visual media, relying only on the information from the self-attention distributions of pre-trained language models (PLMs). With extensive experiments and analyses, we show that 1) our zero-shot approach is superior to a reasonable baseline that adopts TF-IDF and that 2) there exist several attention heads in PLMs specialized for emphasis selection, confirming that PLMs are capable of recognizing important words in sentences.