AI2 is developing a large language model called AI2 OLMo

The Allen Institute for AI has announced embarking on creating an open, state-of-the-art generative language model called AI2 OLMo (Open Language Model). With 70 billion parameters, it will be comparable in scale to other state-of-the-art large language models and is expected in early 2024. However, it is a uniquely open language model intended to benefit the research community by providing access and education around all aspects of model creation. AI2 is developing OLMo in collaboration with AMD and CSC, using the new GPU portion of the all-AMD processor powered LUMI pre-exascale supercomputer. The data and the code used to create the data will be available, as well as the model, the training code, the training curves, and evaluation benchmarks. The OLMo project will include more than just building an open language model — the platform will allow the research community to take each component created and either use it themselves or seek to improve it. Everything created for OLMo will be openly available, documented, and reproducible, with very limited exceptions and under suitable licensing.

The artifacts released as part of the OLMo project will include training data, code, model weights, intermediate checkpoints, and ablations. A release strategy for the model and its artifacts is in development as part of the project. AI2 also plans to build a demo and release interaction data from consenting users.

OLMo project leader and Senior Director of NLP Research at AI2 Hannaneh Hajishirzi says, "With the scientific community in mind, OLMo will be purpose-built to advance the science of language models." The first language model especially created for comprehension and discovery in science is OLMo.

The OLMo model and API will be a powerful new resource for the broader community to better understand and participate in the generative AI revolution. AI2 welcomes support and partnership from organizations aligned with their values of AI for the common good and invested in building responsible, beneficial artificial intelligence technologies.

Noah Smith, an OLMo project lead and a Senior Director of NLP Research at AI2, said, “OLMo will be something special. In a landscape where many are rushing to cash in on the business potential of generative language models, AI2 has… made a commitment to creating a language model that will serve the scientific community and the world more broadly.”

The creation of AI2 OLMo is an important step towards increasing the diversity of perspectives and pace of improvement in language model development. It is also a significant move towards building responsible, beneficial artificial intelligence technologies. The Allen Institute for AI is well-positioned to build a language model that is truly open and uniquely valuable to the AI research community, as a transparent, collaborative, nonprofit institution.