Gpt-neo For Commonsense Reasoning -- A Theoretical And Practical Lens

Kashyap Rohan, Kashyap Vivek, P. Narendra C.. Arxiv 2022

[Paper]    
Attention Mechanism Fine Tuning GPT Model Architecture Pretraining Methods Security Training Techniques

Recent work has demonstrated substantial gains in pre-training large-language models (LLMs) followed by supervised fine-tuning on the downstream task. In this paper, we evaluate the performance of the GPT-neo model using \(6\) commonsense reasoning benchmark tasks. We aim to examine the performance of smaller models using the GPT-neo models against several larger model baselines such as GPT-\(3\), Llama-\(2\), MPT and Falcon. Upon fine-tuning with the appropriate set of hyperparameters, our model achieves competitive accuracy on several tasks. We also investigate and substantiate our results using attention-head visualization to better understand the model performance. Finally, we conduct various robustness tests using various methods to gauge the model performance under numerous settings.

Similar Work