Our latest model advances bring stronger capabilities to building and migrating software in low-resource programming languages like COBOL, Fortran, and Visual Basic. Compared to languages like Python and JavaScript, these languages have less code and documentation available for models to learn from, yet they power critical global infrastructure and core services. COBOL powers over 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions, while Fortran is used for weather forecasting and nuclear reactor safety analysis. Advancing AI capabilities in these languages helps teams maintain and modernize the software that essential services rely on.
This work stems from feedback from developers and enterprise customers who reported that code generation models often struggle with these languages. Maintaining and modernizing this software can be exceptionally challenging: documentation may be incomplete, original developers may have moved on, and critical business rules often reside solely within the code. Whether teams are patching bugs or migrating to another language, they must ensure that the behaviors their systems depend on remain unchanged.
To measure progress, we developed two evaluation benchmarks. MultiLangBench tests whether models can build an SQLite-compatible database engine from scratch across 20 different programming languages. Our COBOL migration evaluation tests whether a model can migrate existing programs to another language while preserving their original behavior. Together, they assess two complementary capabilities: building new software and preserving the functional behavior of existing systems.
Our latest models demonstrate significant improvements across both evaluations, delivering substantially higher accuracy while reducing costs.
On MultiLangBench, GPT‑6 Astra achieved an average test pass rate of 87.4% across 20 programming languages—a dramatic increase from GPT‑5.6 Sol’s 26.8%—showing progress across every tested language. GPT‑6 Luna also outperformed GPT‑5.6 Sol, reaching a 40.3% pass rate while reducing estimated API costs by approximately 78%.
In our COBOL migration evaluation, GPT‑6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared to just 5.4% for GPT‑5.6 Sol under the same setting, with a 39% reduction in cost. At higher reasoning effort, accuracy reached 97.9%, nearly saturating the benchmark.
Taken together, these results show that our models have improved substantially in handling a broad range of programming languages, while becoming significantly more cost-effective.
1. Building SQLite from scratch (MultiLangBench)
MultiLangBench requires models to build an SQLite-compatible database engine in a specified programming language. Models must structure the codebase, manage state, handle errors, and add new features while preserving existing functionality.
To compare implementations across languages, we use database files as a common interface. Candidate programs accept an initial database and an SQL query, then produce a modified database. We use the reference SQLite implementation to verify that the generated database contains the expected contents. This allows us to apply the exact same test cases regardless of the language used to build the engine.
We adapted tests from SQLite’s existing test suite to fit this format. The evaluation measures how much working functionality a model can build across an extended coding session, reporting both test pass rates and API costs.
At maximum reasoning effort, GPT‑6 Astra reached an average test pass rate of 87.4% across 20 languages, up significantly from GPT‑5.6 Sol’s 26.8%. Performance gains spanned every tested language, including COBOL (from 10.9% to 72.1%), OCaml (from 25.6% to 85.8%), Fortran (from 21.4% to 83.5%), and Visual Basic (from 27.5% to 89.4%). Across the 20 languages, Astra scored above 83% on 19 of them, with COBOL remaining the most challenging at 72%.
GPT‑6 Luna offers a lower-cost alternative, also outperforming GPT‑5.6 Sol across all tested languages. At maximum reasoning effort, it achieved an average test pass rate of 40.3%. Gains included COBOL (from 10.9% to 17.1%), OCaml (from 25.6% to 34.1%), Fortran (from 21.4% to 31.4%), and Visual Basic (from 27.5% to 41.6%). These performance improvements come with an estimated average API cost of roughly $0.77 per sample—an approximately 78% reduction compared to $3.48 for GPT‑5.6 Sol.
C C# C++ COBOL Common Lisp Fortran Go Java JavaScript Lean OCaml PHP PowerShell Python Ruby Rust Scala Swift Visual Basic Zig
We have open-sourced the MultiLangBench evaluation harness and test suite on GitHub, along with per-language test results. Developers can inspect the benchmark’s specific tests, evaluate performance in their target languages, and identify gaps critical to their practical workflows.
2. Migrating COBOL codebases
Our second evaluation tasks models with migrating COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving the exact functionality of each program.
A successful migration requires understanding both the program’s business logic and COBOL-specific language behaviors. COBOL programs frequently handle financial transactions and business records, where nuances such as rounding rules and missing-record handling reflect specific business requirements. For example, COBOL’s fixed-point arithmetic truncates excess decimal places unless rounding is explicitly specified. If a rewritten program switches to standard rounding, it will produce different financial results—even if it runs without errors.
To test whether these behaviors are preserved, we evaluate models against held-out tests that require byte-for-byte parity on observable outputs. A migration is only considered successful if all tests pass. While this behavioral fidelity is invaluable for teams maintaining and modernizing legacy systems, applying it to production software still requires expert review and rigorous testing against the system’s business rules and dependencies.
In this evaluation, GPT‑6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared to just 5.4% for GPT‑5.6 Sol under the same setting, at only 61% of the API cost. At higher reasoning effort, accuracy reached 97.9%, bringing the benchmark close to saturation. These results demonstrate that for teams modernizing mission-critical business software, code migration is becoming substantially more accurate and economical.