OpenAI says GPT-6 models improve low-resource language coding

OpenAI introduced two software evaluations that it says show its GPT-6 models can build and migrate code more reliably in programming languages that appear less often in training data. On its new MultiLangBench test, the company said GPT-6 Astra averaged an 87.4% pass rate across 20 languages at maximum reasoning effort, up from 26.8% for GPT-5.6 Sol.

The evaluations focus on two parts of legacy-software work: building a SQLite-compatible database engine in a specified language and translating COBOL programs without changing their observable behavior. OpenAI warned that production use still requires expert review and testing against each system’s business rules and dependencies.

What MultiLangBench measures

MultiLangBench gives a coding model an initial database and a query, then asks it to implement a database engine in one of 20 programming languages. A real SQLite installation validates the database produced by the model’s implementation by running a second query and comparing its output with the expected result.

The public MultiLangBench repository says the benchmark targets SQLite 3.40.1 and converts 24,000 of roughly 51,000 SQLite TCL tests. OpenAI recommends a 272-test subset for routine evaluations. Mismatches, crashes, 60-second timeouts and implementations in the wrong language all count as failures. Internet access is prohibited.

OpenAI said Astra improved on GPT-5.6 Sol in every language tested. The company reported gains from 10.9% to 72.1% in COBOL, 25.6% to 85.8% in OCaml, 21.4% to 83.5% in Fortran and 27.5% to 89.4% in Visual Basic. Astra exceeded 83% in 19 languages, OpenAI said, with COBOL the exception at about 72%.

The company also reported that GPT-6 Luna averaged a 40.3% pass rate at maximum reasoning effort and an estimated API cost of $0.77 per sample. OpenAI put GPT-5.6 Sol at 26.8% and $3.48 per sample, making Luna’s estimated cost about 78% lower.

Testing behavior-preserving COBOL migration

OpenAI’s separate COBOL evaluation asks models to translate programs into C++, Java, Kotlin, Python or Rust. Hidden tests compare observable outputs byte for byte, and a migration succeeds only if every test passes. The standard is designed to catch changes that compilation alone can miss, including differences in fixed-point arithmetic, rounding and missing-record handling.

OpenAI said GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting, while using 61% of the estimated API cost. The company also reported 97.9% accuracy at a higher reasoning effort, but did not identify the exact setting or its corresponding cost.

The released material leaves important limits on independent scrutiny. OpenAI has not published enough detail to reproduce the COBOL evaluation, including the size and provenance of its program set and hidden tests. For MultiLangBench, the company did not disclose the full agent prompt, run count, variance, coding-session budget or whether every published language score used the 272-test subset or all 24,000 converted tests. The supplied research identified no independent reproduction of either set of scores.

MultiLangBench also measures a bounded database implementation, not reliability across production repositories, business-specific dependencies or security requirements. IBM’s overview of COBOL modernization notes that translation alone does not resolve platform integration, data architecture, runtime replacement or transaction-processing integrity.