IBM and Kaggle have launched a public ScarfBench leaderboard. The benchmark checks whether coding agents can migrate enterprise Java applications across Jakarta EE, Quarkus and Spring, with the migrated apps still building, starting and behaving as expected. Kaggle reproduced the evaluation on its platform, sponsored the compute, and will maintain the benchmark and add new models as they are released. Public model submission is not available at launch but is planned.
The evaluation covered 204 migration tasks across ten model and harness pairings. Gemini 4 Argon with Antigravity led on the smoke-test metric at 75.2%, while Claude Opus 5.5 with Claude Code had the highest all-tests-pass rate at 72.1%. The results measure each model paired with its coding agent harness, not the underlying model alone.
The largest gap was between code that compiles and code that behaves correctly. Eight of the ten configurations compiled 97% or more of solvable tasks, yet smoke-test scores spread across more than 50 points. Of 745 classified failing runs, about 72% were runtime behavioral failures, such as lost routes, HTTP 404 errors and content mismatches.
Why it matters if you build agents
Teams automating framework migrations should not treat a successful build as proof that an application works, since runtime route, HTTP and startup checks caught most failures in this evaluation. The leaderboard gives a repeatable way to compare agent configurations on migration work. Its cost data shows wide differences in spend per solved task, so compare behavior and cost together before choosing an agent for modernization.
Get stories like this every weekday
One short email with the day's agent tooling news.