OpenAI published two coding evaluations on 19 October, reporting large gains for its GPT-6 models on COBOL, Fortran and Visual Basic. One test asks models to build a database engine in a specified language; the other asks them to rewrite existing COBOL programs without changing their behaviour. The work addresses languages that OpenAI says have less code and documentation available for model training than Python or JavaScript, but remain in use in banking and other essential systems.
Key points
- OpenAI reports an 87.4 per cent average test pass rate for GPT-6 Astra across 20 languages on MultiLangBench, against 26.8 per cent for GPT-5.6 Sol.
- On its separate COBOL migration test, OpenAI reports 97.9 per cent accuracy for GPT-6.1 Sol at higher reasoning effort.
- The database test checks files against SQLite; the migration test requires rewritten programs to pass every hidden test with identical observable outputs.
- GPT-6 Luna scored lower than Astra on MultiLangBench, while OpenAI estimates its average API cost at $0.77 per sample.
MultiLangBench tests a database built in 20 languages
MultiLangBench gives a model an extended coding task: build an engine compatible with SQLite in one of 20 programming languages. According to OpenAI, the model has to organise the code, keep track of the database’s state, handle errors and add features without breaking earlier ones. That is a different task from producing a short piece of code that looks plausible on inspection.
OpenAI tests the resulting program by giving it a starting database file and an SQL query. The program writes a changed file, which the reference SQLite implementation checks for the expected contents. OpenAI adapted tests from SQLite’s existing test suite for this method, so the same kind of output can be checked regardless of the language used to write the engine. Its reported score is the share of those tests passed, rather than a judgement of the code’s appearance.
At maximum reasoning effort, GPT-6 Astra had an average pass rate of 87.4 per cent across the 20 languages, compared with 26.8 per cent for GPT-5.6 Sol. Its reported COBOL result rose from 10.9 to 72.1 per cent, Fortran from 21.4 to 83.5 per cent, OCaml from 25.6 to 85.8 per cent, and Visual Basic from 27.5 to 89.4 per cent. Astra scored above 83 per cent in 19 languages, OpenAI says, with COBOL its hardest at 72 per cent.
OpenAI says COBOL supports more than 40 per cent of online banking systems, 80 per cent of in-person credit card transactions and 95 per cent of ATM transactions. IBM estimates those same shares, Interesting Engineering reported. OpenAI also identifies weather forecasting and nuclear reactor safety analysis as uses of Fortran. Its developers and enterprise customers had told the company that coding models often struggled with these languages.
COBOL migration requires matching every hidden test
OpenAI’s second evaluation starts with existing COBOL programs and asks models to translate them into C++, Java, Kotlin, Python or Rust. Here the task is to keep the old program’s behaviour, including details that may be easy to change accidentally. OpenAI says documentation can be incomplete and business rules can reside only in the original code, making the program itself an important account of what a replacement must do.
The test checks observable outputs byte for byte and counts a migration as successful only when it passes every hidden test. OpenAI gives decimal arithmetic as an example of the difficulty: COBOL discards excess digits in fixed-point calculations unless rounding is explicitly requested. A translated program that rounds instead could run without an error while producing a different financial result.
Paying by card could depend on a calculation of that kind. If a rewrite rounded digits that the original program would discard, the amount it produced could change even though the replacement ran without errors.
On this evaluation, OpenAI reports 86.6 per cent accuracy for GPT-6.1 Sol at low reasoning effort, against 5.4 per cent for GPT-5.6 Sol, while saying the newer model cost 39 per cent less. At higher reasoning effort, its reported accuracy reached 97.9 per cent. Those are results for translations of the programs in OpenAI’s evaluation under its all-tests-must-pass rule.
Interesting Engineering said developers would still need to check rewritten applications against real workloads and dependencies not captured by the test. That work matters particularly where a calculation or a rule survives in code rather than in separate documentation, as OpenAI describes for the systems behind its migration task.
GPT-6 Luna costs less per MultiLangBench sample
OpenAI also tested GPT-6 Luna on MultiLangBench at maximum reasoning effort. It reports a 40.3 per cent average pass rate across the 20 languages, with improvements over GPT-5.6 Sol in each one. Its reported COBOL pass rate was 17.1 per cent and its Visual Basic rate 41.6 per cent, well below the corresponding Astra scores.
OpenAI estimates Luna’s average API cost at $0.77 per sample, compared with $3.48 for GPT-5.6 Sol, approximately 78 per cent lower. For a company deciding which model to test on its own code, the published Luna figures offer a cost and pass-rate comparison with the older model; OpenAI’s higher Astra pass rate comes from a separate run at maximum reasoning effort.
OpenAI says it is releasing the MultiLangBench evaluation setup, test suite and results for each language on GitHub, where developers can inspect the tests and the language-level results.
