MMLU tests whether an AI model can answer a multiple-choice question about organic chemistry. HumanEval tests whether it can write a function that reverses a linked list. These benchmarks measure what models know. SWE-Bench Verified measures what they can do — specifically, whether they can be given a real GitHub issue, understand the bug or feature request, navigate the codebase, and produce a patch that passes the project's existing test suite and fixes the reported problem.
The distinction matters more than the AI community has acknowledged. Most benchmarks measure knowledge recall and reasoning in a controlled environment. They are SAT-style tests: the problem is defined, the format is consistent, the answer is known. They are useful for tracking model capability trends. They are nearly useless for predicting whether a model will be economically useful in a production software engineering environment, where problems are messy, codebases are large, and the right answer often requires understanding 15 files of context and a five-year-old design decision documented in a single comment.
SWE-Bench Verified — the filtered, higher-quality subset released by the Princeton NLP group in mid-2024 after finding that the original SWE-bench contained ambiguous or ungradable tasks — changes the evaluation paradigm. The agent gets an issue description. It gets access to the repository. It must produce a patch. The evaluation is binary: did the patch fix the issue and not break anything else? This is not a knowledge test. It's a task completion test. The distinction is the same as the distinction between passing the written driver's test and actually driving a car through downtown traffic without hitting anything.
The benchmark scores have been climbing fast enough to warrant attention from people who are not AI researchers. In early 2024, the best model scored roughly 13% on SWE-Bench Verified — meaning it could autonomously resolve about one in eight real-world software issues. By mid-2025, the best agentic systems (models plus scaffolded tool use) were scoring above 40%. By late 2025, several systems crossed 50%. The rate of improvement is roughly 2-3x per year. If the trend continues — and there is no reason to believe it won't, given the pace of model capability improvement and agentic infrastructure investment — autonomous software engineering agents that can handle the majority of routine bugs and small features will be economically viable within 2-3 years.
The economic implications are not "software engineers are obsolete." They are "the marginal cost of fixing a routine bug or adding a small feature is approaching zero." This changes the unit economics of every company that employs software engineers. It changes the pricing models of every company that sells software engineering tools. It changes the competitive dynamics of every industry where software development is a cost center — which is every industry.
The specific bet here: the companies that own the interface for AI-assisted software engineering — the agentic coding platforms, the IDE replacements, the code review automation tools — will capture a meaningful share of the value created by this capability improvement. The model providers (Anthropic, OpenAI, Google DeepMind) will capture another share. The existing developer tools incumbents that do not own an AI-native interface will see their pricing power erode as the cost of producing software declines. The benchmark tells us this is coming. The rate of improvement tells us it's coming faster than most software company valuations assume.
SWE-Bench Verified is not a perfect benchmark. Real software engineering includes ambiguous requirements, cross-team coordination, architectural decisions that have 5-year consequences, and the judgment to know when a hack is acceptable and when it isn't. The benchmark doesn't capture any of that. But it captures enough — the ability to understand intent, navigate unfamiliar code, and produce correct changes — that the trend line is the signal. The absolute score matters less than the slope. The slope is steep.