01.AI's Yi-Coder delivers state-of-the-art coding performance in a small, open-source package

01.AI, an AI startup based in China and a strong contender in the Chinese market looking to grow its global user base by offering its models under open-source licenses, has released Yi-Coder, a model family specializing in coding proficiency. Yi-Coder comes in 1.5B and 9B sizes, each with base and instruction-tuned versions. Yi-Coder 9B resulted from taking another 01.AI model, Yi-9B, and training it with 2.4T high-quality tokens from a GitHub code corpus and data filtered from the CommonCrawl spanning 52 programming languages. It also features a large 128K-token context window to maximize code understanding and generation.

As frequently seen with other models, Yi-Coder 9B is touted as delivering state-of-the-art performance in its size class. Some of the most remarkable results include:

• 23.4% pass rate on LiveCodeBench, meaning the model outperforms larger models, including DeepSeek-Coder-33B-Instruct and CodeLLama-34B-Instruct.
• 85.4% pass rate on HumanEval and 73.8% on MBPP, demonstrating its strong performance in basic code and reasoning capabilities.
• First open-source code LLM to surpass 50% accuracy on CRUXEval-O.

Yi-Coder also excels in code editing, completion, and (program-aided) mathematical reasoning tasks. It demonstrated strong performance in the Aider LLM Leaderboards and CrossCodeEval benchmark. The model's ability to handle long contexts was tested with a "Needle in the code" test, where a customized function is inserted in a random spot in a long codebase, to see if the model can successfully reproduce the function after processing the codebase. Yi-Coder 9B scored 100% accuracy in the evaluation, demonstrating its advanced retrieval capabilities.

Yi-Coder is now available for developers to integrate into their projects, potentially transforming software development processes. For more information, visit the Yi-Coder README or join the discussion on Discord.