OpenAI's unreleased new model may be showing slower rates of progress than its predecessors
Reports suggest OpenAI's new Orion model shows modest improvements over GPT-4, an unexpected development that led the company to accelerate the exploration of alternative enhancement strategies.
According to several reports, the latest installment of OpenAI's GPT models, codenamed Orion, exceeds the performance of GPT-4. However, the leap between these models is smaller than the one between GPT-3 and GPT-4, so much so, that according to some employees who tested the model, Orion may not show reliable improvements over GPT-4 in domain-specific tasks such as coding.
Expectedly, many are interpreting this event as disproving the famous 'scaling laws' that became part and parcel of the generative AI hype. Rather than laws in the strict sense of the word, the so-called scaling 'laws' were projections based on observed model behavior over time. Still, the scaling laws seemed to be holding their ground so well, that the whole industry built around deep learning optimistically expected them to be unbreakable: for all anyone knew, as long as models were provided with endless streams of fresh data and ever-increasing amounts of compute, artificial general intelligence (AGI) could be unlocked in the blink of an eye.
The reports state that, in response to this turn of events, OpenAI has tasked a foundations team with researching and implementing novel approaches to improving performance, such as training Orion with synthetic data generated by other LLMs and enhancing models with post-training improvements based on human feedback. The new o1 model may even be the company's first attempt to test how well post-training enhancements and scaling resources during inference, such as giving the model more processing time and resources to 'think' before it delivers an output, work in real-world use cases.
The situation described in the reports is thoroughly consistent with OpenAI CEO Sam Altman's recent claims in a Reddit AMA that the company is not planning to release a model that could be construed as GPT-5 by any measure, and that OpenAI was choosing to focus on o1 and its successors instead. Given the additional demands models like o1 could place on OpenAI's infrastructure, it is also no wonder the company is rushing its search for alternatives to Nvidia's coveted chips, which has led it to partner with Broadcom and TSMC to design and manufacture an in-house chip.
