Transformer Grammars: Augmenting Transformer Language Models With Syntactic Inductive Biases At Scale

Sartran Laurent, Barrett Samuel, Kuncoro Adhiguna, Stanojević Miloš, Blunsom Phil, Dyer Chris. Arxiv 2022

[Paper]    
Attention Mechanism Ethics And Bias Language Modeling Model Architecture Pretraining Methods Transformer

We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism – one that is independent of composed syntactic representations – plays an important role in current successful models of long text.

Similar Work