Map-neo: Highly Capable And Transparent Bilingual Large Language Model Series

Zhang Ge, Qu Scott, Liu Jiaheng, Zhang Chenchen, Lin Chenghua, Yu Chou Leuang, Pan Danny, Cheng Esther, Liu Jie, Lin Qunshu, Yuan Raven, Zheng Tuney, Pang Wei, Du Xinrun, Liang Yiming, Ma Yinghao, Li Yizhi, Ma Ziyang, Lin Bill, Benetos Emmanouil, Yang Huan, Zhou Junting, Ma Kaijing, Liu Minghao, Niu Morry, Wang Noah, Que Quehry, Liu Ruibo, Liu Sine, Guo Shawn, Gao Soren, Zhou Wangchunshu, Zhang Xinyue, Zhou Yizhi, Wang Yubo, Bai Yuelin, Zhang Yuhan, Zhang Yuxiang, Wang Zenith, Yang Zhenzhu, Zhao Zijian, Zhang Jiajun, Ouyang Wanli, Huang Wenhao, Chen Wenhu. Arxiv 2024

[Paper]    
Ethics And Bias GPT Model Architecture Tools Training Techniques

Large Language Models (LLMs) have made great strides in recent years to achieve unprecedented performance across different tasks. However, due to commercial interest, the most competitive models like GPT, Gemini, and Claude have been gated behind proprietary interfaces without disclosing the training details. Recently, many institutions have open-sourced several strong LLMs like LLaMA-3, comparable to existing closed-source LLMs. However, only the model’s weights are provided with most details (e.g., intermediate checkpoints, pre-training corpus, and training code, etc.) being undisclosed. To improve the transparency of LLMs, the research community has formed to open-source truly open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training corpus and training code) are being provided. These models have greatly advanced the scientific study of these large models including their strengths, weaknesses, biases and risks. However, we observe that the existing truly open LLMs on reasoning, knowledge, and coding tasks are still inferior to existing state-of-the-art LLMs with similar model sizes. To this end, we open-source MAP-Neo, a highly capable and transparent bilingual language model with 7B parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the first fully open-sourced bilingual LLM with comparable performance compared to existing state-of-the-art LLMs. Moreover, we open-source all details to reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning pipeline, checkpoints, and well-optimized training/evaluation framework are provided. Finally, we hope our MAP-Neo will enhance and strengthen the open research community and inspire more innovations and creativities to facilitate the further improvements of LLMs.

Similar Work