GPT-6 Astra, OpenAI's newest frontier model, posts an ARC-AGI-3 score that depends on which evaluation harness runs it: at maximum reasoning effort it scores 62.71% under ARC Prize's Standard harness and 98.55% under OpenAI's Provider Adapter, a 36-point gap at the same setting, per ARC Prize's results page for GPT-6 Astra.
The two scores reflect identical model weights behind the same API, and ARC Prize's rules allow both configurations. Our read is that the disparity is therefore a harness effect rather than a capability gap; the rest of this piece is the evidence for that.
What survives the argument over which number is real is the ranking underneath it.
| Model | ARC-AGI-3 score |
|---|---|
| Astra | 62.7% |
| Claude Opus 5 | 30.2% |
| GPT-5.6 Sol | 7.8% |
| Opus 4.8 | under 2% |
| GPT-5.6 Terra | under 2% |
| GPT-5.5 | under 2% |
| Gemini 3.1 Pro | under 2% |
| Grok 4.5 | under 2% |
Source: BenchLM.ai's ARC-AGI-3 leaderboard, which does not label a harness. Astra's 62.7% is its Standard-harness figure on ARC Prize's own results page, cited above; the rest of the field is as BenchLM lists it.
The harness, not the model
ARC-AGI-3 is not a static test set. It is a turn-based interactive benchmark in which an agent explores unfamiliar 2D game environments across many calls, inferring rules from sparse feedback and building an internal model of how the environment behaves before it can act on it. The determinant of score, ARC Prize found, is what happens to the agent's understanding between those calls.
The Standard harness "enables a model to carry forward notes it chooses to keep with it throughout the environment," per ARC Prize's description of Astra's results. What it does not do, on that description, is preserve anything else between calls. The Provider Adapter harness instead "preserves opaque reasoning state between requests and uses compaction for longer conversations, allowing the model to reuse prior work," the same source states. Compaction is a technique for condensing prior reasoning history so it fits within context limits without being discarded. Under the Standard harness, a model that spent forty calls building a hypothesis about a puzzle's rules has to re-derive that hypothesis, at least in part, on call forty-one. Under the Provider Adapter, it doesn't.
Within the Provider Adapter configuration, reasoning effort turns out to matter comparatively little. Whatever inference-time compute buys a model on this benchmark, it buys less than persistent state does.
None of this is unique to Astra. OpenAI demonstrated the identical mechanism five weeks before Astra's launch, publishing a study showing that enabling retained reasoning and compaction for GPT-5.6 Sol, the previous model, unchanged, took it from 13.3% to 38.3% on ARC-AGI-3's public task set while cutting output tokens sixfold, according to OpenAI's own research post on the effect. OpenAI's framing of that result doubles as an admission about the one that followed: "benchmarks rarely measure AI models in isolation," the post states. "They also measure less visible choices about API settings, harness design, and prompting."
Every generation of this test has reset the field to near zero
The ranking above is a ranking within a benchmark that almost nothing passes. Five of the eight models listed score under 2%, and the third-placed model manages 7.8%, per BenchLM.ai's ARC-AGI-3 leaderboard.
When ARC-AGI-3 was released, it did not merely rank the field low. It erased it. "Humans score 100%. Frontier AI scores 0.51%," ARC Prize wrote in its launch announcement. Its own follow-up analysis of the two leading models put GPT-5.5 at 0.43% and Opus 4.7 at 0.18% on the semi-private set, per ARC Prize's analysis of GPT-5.5 and Opus 4.7. OpenAI's frontier model at the time scored four tenths of one percent on the benchmark its successor now leads.
That is the pattern the series has produced at each generation, not a quirk of this one. ARC-AGI-1 was effectively saturated, with multiple systems above 85%. On ARC-AGI-2, ARC Prize reports that "pure LLMs score 0% on ARC-AGI-2, and public AI reasoning systems achieve only single-digit percentage scores," against a human panel that solved 100% of the tasks, per ARC Prize's ARC-AGI-2 announcement. The same page puts o3-preview-low at 75.7% on ARC-AGI-1 and 4% on ARC-AGI-2: one system, one generation apart, down roughly 72 points.
Astra's own numbers carry a milder version of the same shape. At maximum reasoning effort it scores 97.5% on ARC-AGI-1, 95.0% on ARC-AGI-2 and 62.71% on ARC-AGI-3's Standard harness, per ARC Prize's per-generation table for Astra. The decline is far shallower than the collapse that met the previous generation of models, which is the strongest thing that can be said for it, and it is still a decline of 35 points across two generations of the same benchmark family.
Our read is that the launch figure, not the leaderboard, is what makes the harness question consequential rather than academic. Within one model generation at one lab, the same benchmark went from erasing the field to being led outright — and a large slice of that range turns out to be recoverable by changing what the evaluation preserves between calls rather than by changing the model. On a test whose scores have twice been reset to near zero, the configuration a vendor evaluates under can be worth more than a generation of model progress. That is an argument for reading the harness before the headline number, not for discounting the benchmark.
What a mature benchmark cannot distinguish
The launch figure and the current leader are separated by months in which ARC-AGI-3 was public. ARC Prize published its per-model analysis of the benchmark in April 2026 and Astra's results in September, so the environments, the scoring and the documented failure modes had been available for roughly five months before the run that now leads, per ARC Prize's analysis of GPT-5.5 and Opus 4.7.
Some of that optimisation is not inferred, it is published. The OpenAI study cited above found two evaluation settings that tripled its previous model's score on this benchmark without changing the model at all. Our read is that this is a lab aiming at a known target, and publishing the aim.
Our read is that this is the limit of what the current number can settle. A score on a benchmark generation that has been public for months is consistent with a model that generalises better, and equally consistent with a lab that has had months to aim at a fixed target; nothing in the score separates the two. What would separate them is day-one performance on a generation nobody has seen — and on the record so far, that has gone the other way every time. The drop from ARC-AGI-1 to ARC-AGI-2 cost the leading system roughly 72 points, and the frontier opened ARC-AGI-3 at half a percent, both per the ARC Prize pages cited above. No model has yet met a new ARC-AGI generation at anything above the floor. Until one does, a leaderboard score on a mature benchmark measures adaptation to a known test at least as much as the capability the test was built to find.
Why the adapter run was also cheaper
At maximum reasoning effort the Provider Adapter run cost $17,332 against $26,098 for the Standard harness, despite scoring 36 points higher, per ARC Prize's results page for GPT-6 Astra. That is consistent with the same mechanism working in both directions: a model that does not have to regenerate its reasoning from scratch at every call also does not have to pay for regenerating it. The sixfold reduction in output tokens OpenAI reported for Sol under the same settings points to where the savings come from, not less work per call, but far less repeated work across calls.
That efficiency argument cuts against treating the cost gap as independent evidence that the adapter run reflects equivalent computational effort to the Standard run. It doesn't have to expend equivalent effort to score higher; it has to avoid wasting the effort it already spent.
What the benchmark's own history says about a 36-point gap
ARC-AGI-3 was built to be hard. The case that it is more than harness engineering rests on a behavior ARC Prize documented independently of the score: Astra "represented game mechanics as logical rules and developed its own domain-specific language shorthand to track state and plan actions," and used fewer actions than the median human tester on 96% of levels, according to ARC Prize's Astra writeup.
Separately, Fortune reported that OpenAI's Astra blog post was published roughly two hours later than its planned 2pm ET slot, and that OpenAI subsequently revised several published evaluation figures after launch, with some of its own numbers rising and at least one competitor's figure lowered. Fortune names the figures. Astra's hallucination rate went from 4.2% to 2% and back to 4.2%; Sol's ExploitBench score went from 5.5% to 11.5%; a competitor's FrontierMath score fell from 87.8% to 78% and settled at 83%. Most directly for this piece, Fortune reports that an embargoed pre-publication draft listed Astra's ARC-AGI-3 score as 98.6%, and that the live blog now reads 99.99%. The sequence, a delayed rollout followed by post-publication edits to a comparative benchmark table, is the kind of detail that makes a self-reported adapter score harder to take at face value regardless of whether the underlying mechanism is legitimate.