MCP-Atlas is becoming the benchmark people quote when they want to say a model is good at tool use. Meta cited it in the Muse Glimmer release, reporting 75.5% [3]. It is a good benchmark, built with more care than most.
It also reports a top score of 44.5% in one place and 82.2% in another [1][2].
Neither figure is wrong. Both come from the people who built it. But if you are comparing two models on the strength of an MCP-Atlas number somebody put in a launch post, you need to know which of those regimes produced it, and almost nobody says.
Where the two numbers come from
Scale's original leaderboard post, published 19 September 2025, scored 1,000 tasks across 40-plus MCP servers and 300-plus tools, with 3 to 6 tool calls per task and distractors included. Scoring was pass or fail on whether the final answer matched ground truth. GPT-5 led at 44.5%, which is 445 of 1,000 tasks. The median model, Kimi-K2, managed 23.9%, and half the field sat between 8% and 38.3% [2].
The arXiv paper, first submitted 31 January 2026 and revised to v3 on 19 May 2026, describes 1,000 expert-written tasks across 36 real MCP servers and 220 tools, with 500 released publicly and 500 held back to protect the leaderboard. Across 20 frontier models from six providers, pass rates reach 82.2% at a 0.75 claim-coverage threshold [1].
Read those carefully and the difference is not a model getting better. Three things changed at once.
The scoring changed, from binary final-answer matching to a claim-level rubric where an answer is scored against atomic factual claims extracted from tool outputs. That rubric deliberately gives credit for alternative tool-call sequences that reach the same grounded result [1]. The threshold is explicit in the paper's number: 82.2% is measured at 0.75 claim coverage, meaning an answer counts if it captures three-quarters of the required claims. And the benchmark itself changed, from 40-plus servers and 300-plus tools to 36 servers and 220 tools, with half the tasks withheld [1][2].
So one number asks "did the model produce the right final answer" and the other asks "did the model's answer cover 75% of the grounded claims". Those are different questions about different task sets. Both are legitimate. They are not comparable, and a bare percentage carries none of that context.
The hard part is discovery, not invocation
One design decision explains why the 2025 numbers were so low, and it is the most realistic thing about the benchmark.
Prompts do not name the server, the tool, or the parameters. The agent has to work out which tools are relevant from the request alone, while the task set deliberately includes semantically similar alternatives as distractors, then compose a multi-step workflow across more than one server [1][2]. Tasks require between 3 and 6 tool calls [2].
That is a different skill from what most tool-use evaluations measure. Give a model a short, curated tool list and ask it to pick one, and you are testing invocation. Give it a realistic registry where four tools plausibly match and the right answer needs three of them in sequence across two servers, and you are testing discovery, selection and orchestration.
Discovery is the part that breaks in production. Nobody deploys an agent with exactly the four tools it needs and no near-duplicates, and MCP's whole premise is that the tool registry grows beyond what a developer curates by hand.
Why the claim-level rubric is the better idea anyway
It is worth being clear that the paper's approach is an improvement, not a softening.
Binary final-answer matching punishes a model for taking a different but valid route to the same place. In a multi-server workflow there are usually several defensible orderings, and grading on exact output makes the benchmark partly a test of formatting agreement. Scoring against atomic claims derived from tool outputs measures whether the model actually got the grounded facts, and it disentangles the result from verbosity and style [1].
The paper is explicit that this was one of three gaps it set out to close, alongside realistic multi-step cross-server workflows and breadth across authentic servers rather than mocks [1].
The catch is that a rubric with a coverage threshold turns one score into a family of scores. Quote 82.2% without saying "at 0.75 coverage" and you have published a number that cannot be reproduced or compared.
What the spread looked like
The 2025 leaderboard is worth reading for its distribution, not just its top line. GPT-5 led at 44.5%. The median model, Kimi-K2, scored 23.9%. Half the field fell between 8% and 38.3% [2].
A benchmark where the best available model fails more than half the tasks, and the median model fails three quarters of them, is either badly designed or measuring something genuinely hard. Given the discovery-plus-orchestration setup, the second reading is the fair one.
The paper's 20-model run reports a three-tier performance structure rather than a smooth gradient [1], which suggests models cluster into capability bands on this task type rather than improving continuously. That is more useful for model selection than a single ranking: it tells you there are a few real choices rather than twenty slightly different ones.
The finding that matters more than either score
Buried in the paper is a result that should change how you debug agents.
The authors built an 11-category diagnostic taxonomy separating tool-call failures from cognitive failures in comprehension, synthesis, parsing and stopping criteria. Applying it, they found that 63.3% of diagnosed failures are cognitive rather than tool-call related [1].
Nearly two thirds of failures are not the model calling the wrong tool, malforming arguments, or fumbling the protocol. The model finds the right tools, calls them correctly, gets valid data back, and then stops too early, or synthesises the results wrongly, or misreads what was being asked.
This inverts where most teams spend their effort. Tool-use debugging usually means better tool descriptions, stricter argument schemas, retries on malformed calls. All useful, all aimed at the 36.7%.
The larger failure is a model that executed its plumbing perfectly and then drew the wrong conclusion, or decided it was finished when it was not. Premature stopping in particular is nearly invisible in logs: the trace shows successful calls and a confident answer. There is no error to catch.
Readers of this site will recognise the shape. This is the same failure that produced three confident "Done" reports over broken pages in the DeepSeek Harness testing, and the same reason consensus in multi-agent debate cannot be read as correctness. The system reports success by its own measure while missing the actual objective.
What this means for the local-model claim
Meta's Muse Glimmer release reported 75.5% on MCP-Atlas [3], which is the number that made a 30B open-weight model look competitive with frontier hosted models on tool use.
I have no reason to doubt it, and the model's benchmark set was reported against Gemma 4 31B and Qwen 3.6 27B rather than against the Scale leaderboard [3]. But 75.5% sits between the two published regimes, and without knowing the threshold and the task split it is not directly comparable to either 44.5% or 82.2%.
That is not an accusation. It is the ordinary consequence of a benchmark evolving while its name stays the same, and it will get worse as more vendors cite it.
The questions to ask
When a model card or a launch post quotes an MCP-Atlas score, four things make it meaningful:
Which task set. The public 500, the full 1,000, or the older 40-plus-server version [1][2]. If the answer is the public 500, the model may have been trained on tasks that are now in the open.
Which scoring. Binary final-answer match or claim coverage [1][2]. These are not close to each other.
Which threshold. If claim coverage, at what level. 82.2% is a 0.75-coverage number [1]. A 0.5-coverage number would be higher and would mean something weaker.
Which failure breakdown. Given that 63.3% of failures are cognitive [1], a headline pass rate tells you very little about whether a model will stop early on your workload. The diagnostic categories are more informative than the score.
If a vendor cannot answer the first three, treat the number as marketing rather than measurement. That is not cynicism about MCP-Atlas, which is among the more carefully built agent benchmarks available. It is a comment on what happens to any benchmark once its name becomes a credential.
The useful takeaway
Run the public 500 yourself if tool use matters to your product [1]. A benchmark you executed, on your own harness, with your own model configuration and your own threshold choice, tells you something a quoted percentage never will.
And when you do, read the failure taxonomy before the score. The score tells you where a model ranks. The 63.3% tells you that most of what goes wrong is not the part you have been instrumenting.
References
[1] Bandi, Dumitru, Hertzberg, Agarwal and others — MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers. Paper
[2] The Scale Research Team — MCP-Atlas Raises the Bar for Agentic Evaluation. Article
[3] Olimpiu Pop, InfoQ — Meta Open-Sources Muse Glimmer. Article