The latest benchmark for AI coding agents, SWE-Bench ProMax, shows a persistent gap in handling large-scale refactoring. Even the best models reach only a 41.2% success rate in this area.
Created by a team at Shanghai Jiao Tong University, Peking University, and Douyin Group, the benchmark focuses on a weakness most evaluations ignore. It includes 170 real-world tasks across seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust. Each task went through multiple review stages to remove flawed tests and ensure sufficient difficulty.
Refactoring requires exact precision. Unlike simpler coding tasks, it allows no room for errors, behavior changes, or irreversible edits. Shane Warden, principal architect at ActiveState, explains that treating source code as plain text restricts how well AI can perform these tasks.
“Engineers who see large language models as text processors approach refactoring as a text-generation problem,” Warden said. “They load an entire codebase into a large context window, prompt the model, and wait for multi-file changes. But token proximity doesn’t ensure the model grasps the structure.”
Vojtěch Pavlík, senior director of technical strategy at SUSE, identifies another issue: attention mechanisms in these models struggle with massive codebases. “For very large models, optimized attention algorithms like Deepseek Sparse Attention become essential,” he said. “Without them, models overlook key details or become confused when the context window fills up.”
Time adds another challenge. Pavlík notes that models often miss race conditions, idempotency, or retry logic—problems that appear when parallel tasks run unpredictably. “Humans operate in a world shaped by time,” he said. “LLMs don’t.”
Most coding benchmarks emphasize speed and predictable outcomes, often excluding large-scale refactoring. Pavlík says this creates a misleading view of AI capabilities. “Many advanced models now memorize answers to common benchmarks,” he said. “The more widely used a benchmark becomes, the less trustworthy its results are.”
SWE-Bench ProMax avoids this problem by focusing on complex, cross-file tasks that resist memorization. The change reflects a broader shift in benchmark design—from measuring general performance to driving targeted improvements. “It shows how benchmarks are changing,” Pavlík said, “from ‘we need to gauge how good the model is’ to ‘we need models to get better at specific tasks.’”
Related: Tech innovators celebrated in 2026 awards
Earlier versions, like SWE-bench Verified, faced criticism for including flawed tests in nearly 60% of unsolved cases. The new benchmark fixes this by manually reviewing test suites to remove overly narrow or broad conditions. This ensures tasks require real problem-solving instead of pattern matching.
Current AI models still can’t fully grasp a large codebase. Pavlík calls this ability “far out of reach.” The issue isn’t just whether AI can refactor code, but whether it can do so without introducing subtle, hard-to-find errors.
Closing this gap will take more than bigger context windows or improved attention. Pavlík suggests a layered approach: mapping the codebase to create detailed specifications, using retrieval-augmented generation to focus on relevant sections, and including human validation. A strong test suite for the original code is also necessary to verify correctness before refactoring starts.
Warden offers a different solution—an architecture where models control deterministic tools instead of generating raw changes. “The most promising work involves models detecting code issues, weighing design trade-offs, and querying Language Server Protocols to analyze structures,” he said. The aim would be to issue small, reversible commands, undoing changes if tests fail.
For now, SWE-Bench ProMax acts as a reality check. While AI coding agents perform well on narrow, defined tasks, large-scale refactoring remains beyond their current abilities. The benchmark doesn’t just point out the problem—it outlines where progress is needed.
If history offers any lesson, future models won’t solve these challenges through sheer scale. They’ll need a better way to work with code—one that sees it as more than text.
Recent security breaches highlight how critical this shift could be for protecting software integrity.
