Same model, four different tools, three times the price: a test of what the wrapper costs you

Here is a result worth knowing before you pick an AI coding tool. The AI tooling company Composio took a single model, DeepSeek V4 Flash, and ran it through four different agent frameworks on the same 30 tasks, using real services like Gmail, GitHub, Slack and Notion. Same model every time. Only the software around it changed.

The success rates came out close. Oh My Pi finished the most tasks at 17 out of 30, Claude Code and Codex sat just behind, and OpenCode trailed slightly at 14 out of 30. Seven of the 30 tasks passed or failed purely because of which framework ran them, which is itself a small warning about how repeatable any of this is.

The gaps that actually mattered were cost and time. OpenCode was cheapest at 0.073 dollars per successful task. Claude Code was the most expensive at 0.195 dollars, nearly three times as much, and yet it was also the fastest at 122 seconds per task, against 272 seconds for the slowest, Oh My Pi. Claude Code got there while using the fewest tool calls and generating the least output text, which is a nice illustration that “efficient” and “cheap” are not the same word.

What is actually going on here

An agent framework, sometimes called a harness, is the layer between you and the model. It decides how your request is phrased, how much of the conversation history gets resent on each step, when to call an external tool, how many times to retry, and when to stop. The model does the thinking, but the framework decides how many times the model gets to think and how much it has to read each time. Since you pay per unit of text processed, that bookkeeping is most of your bill.

This is why comparing models on a leaderboard only tells you part of the story. A benchmark score describes a model under one specific setup. Your actual cost and speed depend just as much on the tool you run it in, and those numbers move by multiples, not percentages.

Some fair caveats. Thirty tasks is a small sample, the test used one model, and success rates in the 14 to 17 range mean roughly half of everything failed regardless of the tool. Composio sells tooling in this space, so read it as a useful data point rather than a neutral verdict. And all four frameworks are moving targets that will have changed by next month.

What this means for you: if you use an AI agent for anything repetitive, the tool is a real variable, not a detail. Run a handful of your own typical tasks through two options and compare the bill and the wall clock, because the difference is large enough to notice. If you are just starting out, do not agonise over the choice, but do turn on whatever cost display your tool offers. The single most useful habit here is knowing what a task costs you before you run it a hundred times.

Sources

• Claude Code is the fastest agent framework but costs nearly three times more, The Decoder
• Composio test results via X