Picking The Underused Heads: A Network Pruning Perspective Of Attention Head Selection For Fusing Dialogue Coreference Information

Liu Zhengyuan, Chen Nancy F.. Arxiv 2023

[Paper]    
Applications Attention Mechanism Efficiency And Optimization Model Architecture Pretraining Methods Pruning Training Techniques Transformer

The Transformer-based models with the multi-head self-attention mechanism are widely used in natural language processing, and provide state-of-the-art results. While the pre-trained language backbones are shown to implicitly capture certain linguistic knowledge, explicitly incorporating structure-aware features can bring about further improvement on the downstream tasks. However, such enhancement often requires additional neural components and increases training parameter size. In this work, we investigate the attention head selection and manipulation strategy for feature injection from a network pruning perspective, and conduct a case study on dialogue summarization. We first rank attention heads in a Transformer-based summarizer with layer-wise importance. We then select the underused heads through extensive analysis, and inject structure-aware features by manipulating the selected heads. Experimental results show that the importance-based head selection is effective for feature injection, and dialogue summarization can be improved by incorporating coreference information via head manipulation.

Similar Work