Amazon may be training an ambitious 2T-parameter LLM

According to people familiar with the project, Amazon is financing efforts to train a 2 trillion parameter large language model that can compete with OpenAI and Google's best offerings. OpenAI has never officially disclosed details on its GPT family of models. However, news site Semafor claimed they had "spoken to eight people familiar with the inside story" and published that GPT-4 had 1 trillion parameters. Since then, the unconfirmed estimate has taken off. If true, it would mean that Amazon's model, codenamed 'Olympus', would be twice as large as GPT-4 and one of the largest models in training.

The report originally appeared in Reuters and details that Rohit Prasad, the former head of Alexa, leads the team behind Olympus. Prasad now reports directly to CEO Andy Jassy. He is unifying the AI efforts across the company by bringing together researchers previously employed at Alexa AI and the Amazon science team to work on training models. According to the two anonymous sources, there isn't a specific timeline for the model's release.

Acquiring enough computing power to train large-sized models is costly. As a result, Amazon recently stated that it "would increase investment in LLMs and generative AI while cutting back on fulfillment and transportation in its retail business." The expectation is that once released, Olympus will join Amazon's smaller model Titan on Amazon Bedrock, the platform available for Amazon Web Services (AWS) users to build and scale generative AI applications. Bedrock allows developers to deploy high-quality proprietary and open-source models such as Anthropic's Claude, Cohere's Command, Stable Diffusion XL, and soon, Meta's Llama 2. Olympus would be the largest model to join the lineup at Bedrock.