Beyond The Imitation Game: Quantifying And Extrapolating The Capabilities Of Language Models
Srivastava Aarohi Shammie, Rastogi Abhinav Shammie, Rao Abhishek Shammie, Shoeb Abu Awal Md Shammie, Abid Abubakar Shammie, Fisch Adam Shammie, Brown Adam R. Shammie, Santoro Adam Shammie, Gupta Aditya Shammie, Garriga-alonso Adrià Shammie, Kluska Agnieszka Shammie, Lewkowycz Aitor Shammie, Agarwal Akshat Shammie, Power Alethea Shammie, Ray Alex Shammie, Warstadt Alex Shammie, Kocurek Alexander W. Shammie, Safaya Ali Shammie, Tazarv Ali Shammie, Xiang Alice Shammie, Parrish Alicia Shammie, Nie Allen Shammie, Hussain Aman Shammie, Askell Amanda Shammie, Dsouza Amanda Shammie, Slone Ambrose Shammie, Rahane Ameet Shammie, Iyer Anantharaman S. Shammie, Andreassen Anders Shammie, Madotto Andrea Shammie, Santilli Andrea Shammie, Stuhlmüller Andreas Shammie, Dai Andrew Shammie, La Andrew Shammie, Lampinen Andrew Shammie, Zou Andy Shammie, Jiang Angela Shammie, Chen Angelica Shammie, Vuong Anh Shammie, Gupta Animesh Shammie, Gottardi Anna Shammie, Norelli Antonio Shammie, Venkatesh Anu Shammie, Gholamidavoodi Arash Shammie, Tabassum Arfa Shammie, Menezes Arul Shammie, Kirubarajan Arun Shammie, Mullokandov Asher Shammie, Sabharwal Ashish Shammie, Herrick Austin Shammie, Efrat Avia Shammie, Erdem Aykut Shammie, Karakaş Ayla Shammie, Roberts B. Ryan Shammie, Loe Bao Sheng Shammie, Zoph Barret Shammie, Bojanowski Bartłomiej Shammie, Özyurt Batuhan Shammie, Hedayatnia Behnam Shammie, Neyshabur Behnam Shammie, Inden Benjamin Shammie, Stein Benno Shammie, Ekmekci Berk Shammie, Lin Bill Yuchen Shammie, Howald Blake Shammie, Orinion Bryan Shammie, Diao Cameron Shammie, Dour Cameron Shammie, Stinson Catherine Shammie, Argueta Cedrick Shammie, Ramírez César Ferri Shammie, Singh Chandan Shammie, Rathkopf Charles Shammie, Meng Chenlin Shammie, Baral Chitta Shammie, Wu Chiyu Shammie, Callison-burch Chris Shammie, Waites Chris Shammie, Voigt Christian Shammie, Manning Christopher D. Shammie, Potts Christopher Shammie, Ramirez Cindy Shammie, Rivera Clara E. Shammie, Siro Clemencia Shammie, Raffel Colin Shammie, Ashcraft Courtney Shammie, Garbacea Cristina Shammie, Sileo Damien Shammie, Garrette Dan Shammie, Hendrycks Dan Shammie, Kilman Dan Shammie, Roth Dan Shammie, Freeman Daniel Shammie, Khashabi Daniel Shammie, Levy Daniel Shammie, González Daniel Moseguí Shammie, Perszyk Danielle Shammie, Hernandez Danny Shammie, Chen Danqi Shammie, Ippolito Daphne Shammie, Gilboa Dar Shammie, Dohan David Shammie, Drakard David Shammie, Jurgens David Shammie, Datta Debajyoti Shammie, Ganguli Deep Shammie, Emelin Denis Shammie, Kleyko Denis Shammie, Yuret Deniz Shammie, Chen Derek Shammie, Tam Derek Shammie, Hupkes Dieuwke Shammie, Misra Diganta Shammie, Buzan Dilyar Shammie, Mollo Dimitri Coelho Shammie, Yang Diyi Shammie, Lee Dong-ho Shammie, Schrader Dylan Shammie, Shutova Ekaterina Shammie, Cubuk Ekin Dogus Shammie, Segal Elad Shammie, Hagerman Eleanor Shammie, Barnes Elizabeth Shammie, Donoway Elizabeth Shammie, Pavlick Ellie Shammie, Rodola Emanuele Shammie, Lam Emma Shammie, Chu Eric Shammie, Tang Eric Shammie, Erdem Erkut Shammie, Chang Ernie Shammie, Chi Ethan A. Shammie, Dyer Ethan Shammie, Jerzak Ethan Shammie, Kim Ethan Shammie, Manyasi Eunice Engefu Shammie, Zheltonozhskii Evgenii Shammie, Xia Fanyue Shammie, Siar Fatemeh Shammie, Martínez-plumed Fernando Shammie, Happé Francesca Shammie, Chollet Francois Shammie, Rong Frieda Shammie, Mishra Gaurav Shammie, Winata Genta Indra Shammie, De Melo Gerard Shammie, Kruszewski Germán Shammie, Parascandolo Giambattista Shammie, Mariani Giorgio Shammie, Wang Gloria Shammie, Jaimovitch-lópez Gonzalo Shammie, Betz Gregor Shammie, Gur-ari Guy Shammie, Galijasevic Hana Shammie, Kim Hannah Shammie, Rashkin Hannah Shammie, Hajishirzi Hannaneh Shammie, Mehta Harsh Shammie, Bogar Hayden Shammie, Shevlin Henry Shammie, Schütze Hinrich Shammie, Yakura Hiromu Shammie, Zhang Hongming Shammie, Wong Hugh Mee Shammie, Ng Ian Shammie, Noble Isaac Shammie, Jumelet Jaap Shammie, Geissinger Jack Shammie, Kernion Jackson Shammie, Hilton Jacob Shammie, Lee Jaehoon Shammie, Fisac Jaime Fernández Shammie, Simon James B. Shammie, Koppel James Shammie, Zheng James Shammie, Zou James Shammie, Kocoń Jan Shammie, Thompson Jana Shammie, Wingfield Janelle Shammie, Kaplan Jared Shammie, Radom Jarema Shammie, Sohl-dickstein Jascha Shammie, Phang Jason Shammie, Wei Jason Shammie, Yosinski Jason Shammie, Novikova Jekaterina Shammie, Bosscher Jelle Shammie, Marsh Jennifer Shammie, Kim Jeremy Shammie, Taal Jeroen Shammie, Engel Jesse Shammie, Alabi Jesujoba Shammie, Xu Jiacheng Shammie, Song Jiaming Shammie, Tang Jillian Shammie, Waweru Joan Shammie, Burden John Shammie, Miller John Shammie, Balis John U. Shammie, Batchelder Jonathan Shammie, Berant Jonathan Shammie, Frohberg Jörg Shammie, Rozen Jos Shammie, Hernandez-orallo Jose Shammie, Boudeman Joseph Shammie, Guerr Joseph Shammie, Jones Joseph Shammie, Tenenbaum Joshua B. Shammie, Rule Joshua S. Shammie, Chua Joyce Shammie, Kanclerz Kamil Shammie, Livescu Karen Shammie, Krauth Karl Shammie, Gopalakrishnan Karthik Shammie, Ignatyeva Katerina Shammie, Markert Katja Shammie, Dhole Kaustubh D. Shammie, Gimpel Kevin Shammie, Omondi Kevin Shammie, Mathewson Kory Shammie, Chiafullo Kristen Shammie, Shkaruta Ksenia Shammie, Shridhar Kumar Shammie, Mcdonell Kyle Shammie, Richardson Kyle Shammie, Reynolds Laria Shammie, Gao Leo Shammie, Zhang Li Shammie, Dugan Liam Shammie, Qin Lianhui Shammie, Contreras-ochando Lidia Shammie, Morency Louis-philippe Shammie, Moschella Luca Shammie, Lam Lucas Shammie, Noble Lucy Shammie, Schmidt Ludwig Shammie, He Luheng Shammie, Colón Luis Oliveros Shammie, Metz Luke Shammie, Şenel Lütfi Kerem Shammie, Bosma Maarten Shammie, Sap Maarten Shammie, Ter Hoeve Maartje Shammie, Farooqi Maheen Shammie, Faruqui Manaal Shammie, Mazeika Mantas Shammie, Baturan Marco Shammie, Marelli Marco Shammie, Maru Marco Shammie, Quintana Maria Jose Ramírez Shammie, Tolkiehn Marie Shammie, Giulianelli Mario Shammie, Lewis Martha Shammie, Potthast Martin Shammie, Leavitt Matthew L. Shammie, Hagen Matthias Shammie, Schubert Mátyás Shammie, Baitemirova Medina Orduna Shammie, Arnaud Melody Shammie, Mcelrath Melvin Shammie, Yee Michael A. Shammie, Cohen Michael Shammie, Gu Michael Shammie, Ivanitskiy Michael Shammie, Starritt Michael Shammie, Strube Michael Shammie, Swędrowski Michał Shammie, Bevilacqua Michele Shammie, Yasunaga Michihiro Shammie, Kale Mihir Shammie, Cain Mike Shammie, Xu Mimee Shammie, Suzgun Mirac Shammie, Walker Mitch Shammie, Tiwari Mo Shammie, Bansal Mohit Shammie, Aminnaseri Moin Shammie, Geva Mor Shammie, Gheini Mozhdeh Shammie, T Mukund Varma Shammie, Peng Nanyun Shammie, Chi Nathan A. Shammie, Lee Nayeon Shammie, Krakover Neta Gur-ari Shammie, Cameron Nicholas Shammie, Roberts Nicholas Shammie, Doiron Nick Shammie, Martinez Nicole Shammie, Nangia Nikita Shammie, Deckers Niklas Shammie, Muennighoff Niklas Shammie, Keskar Nitish Shirish Shammie, Iyer Niveditha S. Shammie, Constant Noah Shammie, Fiedel Noah Shammie, Wen Nuan Shammie, Zhang Oliver Shammie, Agha Omar Shammie, Elbaghdadi Omar Shammie, Levy Omer Shammie, Evans Owain Shammie, Casares Pablo Antonio Moreno Shammie, Doshi Parth Shammie, Fung Pascale Shammie, Liang Paul Pu Shammie, Vicol Paul Shammie, Alipoormolabashi Pegah Shammie, Liao Peiyuan Shammie, Liang Percy Shammie, Chang Peter Shammie, Eckersley Peter Shammie, Htut Phu Mon Shammie, Hwang Pinyu Shammie, Miłkowski Piotr Shammie, Patil Piyush Shammie, Pezeshkpour Pouya Shammie, Oli Priti Shammie, Mei Qiaozhu Shammie, Lyu Qing Shammie, Chen Qinlang Shammie, Banjade Rabin Shammie, Rudolph Rachel Etta Shammie, Gabriel Raefer Shammie, Habacker Rahel Shammie, Risco Ramon Shammie, Millière Raphaël Shammie, Garg Rhythm Shammie, Barnes Richard Shammie, Saurous Rif A. Shammie, Arakawa Riku Shammie, Raymaekers Robbe Shammie, Frank Robert Shammie, Sikand Rohan Shammie, Novak Roman Shammie, Sitelew Roman Shammie, Lebras Ronan Shammie, Liu Rosanne Shammie, Jacobs Rowan Shammie, Zhang Rui Shammie, Salakhutdinov Ruslan Shammie, Chi Ryan Shammie, Lee Ryan Shammie, Stovall Ryan Shammie, Teehan Ryan Shammie, Yang Rylan Shammie, Singh Sahib Shammie, Mohammad Saif M. Shammie, Anand Sajant Shammie, Dillavou Sam Shammie, Shleifer Sam Shammie, Wiseman Sam Shammie, Gruetter Samuel Shammie, Bowman Samuel R. Shammie, Schoenholz Samuel S. Shammie, Han Sanghyun Shammie, Kwatra Sanjeev Shammie, Rous Sarah A. Shammie, Ghazarian Sarik Shammie, Ghosh Sayan Shammie, Casey Sean Shammie, Bischoff Sebastian Shammie, Gehrmann Sebastian Shammie, Schuster Sebastian Shammie, Sadeghi Sepideh Shammie, Hamdan Shadi Shammie, Zhou Sharon Shammie, Srivastava Shashank Shammie, Shi Sherry Shammie, Singh Shikhar Shammie, Asaadi Shima Shammie, Gu Shixiang Shane Shammie, Pachchigar Shubh Shammie, Toshniwal Shubham Shammie, Upadhyay Shyam Shammie, Shyamolima Shammie, Debnath, Shakeri Siamak, Thormeyer Simon, Melzi Simone, Reddy Siva, Makini Sneha Priscilla, Lee Soo-hwan, Torene Spencer, Hatwar Sriharsha, Dehaene Stanislas, Divic Stefan, Ermon Stefano, Biderman Stella, Lin Stephanie, Prasad Stephen, Piantadosi Steven T., Shieber Stuart M., Misherghi Summer, Kiritchenko Svetlana, Mishra Swaroop, Linzen Tal, Schuster Tal, Li Tao, Yu Tao, Ali Tariq, Hashimoto Tatsu, Wu Te-lin, Desbordes Théo, Rothschild Theodore, Phan Thomas, Wang Tianle, Nkinyili Tiberius, Schick Timo, Kornev Timofei, Tunduny Titus, Gerstenberg Tobias, Chang Trenton, Neeraj Trishala, Khot Tushar, Shultz Tyler, Shaham Uri, Misra Vedant, Demberg Vera, Nyamai Victoria, Raunak Vikas, Ramasesh Vinay, Prabhu Vinay Uday, Padmakumar Vishakh, Srikumar Vivek, Fedus William, Saunders William, Zhang William, Vossen Wout, Ren Xiang, Tong Xiaoyu, Zhao Xinran, Wu Xinyi, Shen Xudong, Yaghoobzadeh Yadollah, Lakretz Yair, Song Yangqiu, Bahri Yasaman, Choi Yejin, Yang Yichi, Hao Yiding, Chen Yifu, Belinkov Yonatan, Hou Yu, Hou Yufang, Bai Yuntao, Seid Zachary, Zhao Zhuoye, Wang Zijian, Wang Zijie J., Wang Zirui, Wu Ziyi. Transactions on Machine Learning Research May/ 2022
[Paper]
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI’s GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit “breakthrough” behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Similar Work