Sourced guide

Kimi K3 beat Fable 5 on cost, not on effort

Fireworks ran a 1,000-task agent benchmark comparing an open and a closed model. The routing result matters more than either model winning.

What happened

Fireworks ran Kimi K3, an open model, against Fable 5, a closed model, on roughly 1,000 agentic tasks. The headline number is cost: routing between the two models was up to 50 times more cost effective than running Fable alone on long agentic loops.

The benchmark harness totaled about 1,030 tasks across five families, run inside real agent loops rather than one-shot prompts. Two families give the clearest picture: a SWE-bench style set of real repository bug fixes (460 tasks), and a legal-agent benchmark with lawyer-graded tasks (120 tasks).

The routing signal

Oracle routing selected K3 for 72 to 96 percent of tasks, depending on the family. That is not K3 winning everything — it is K3 being the right choice for most of the work, most of the time, when the routing decision is made per task rather than fixed in advance.

Of 89 terminal tasks, K3 had 11 solo wins to Fable's 7. K3 worked noticeably harder to get there: on the SWE-style tasks it averaged roughly 55 turns and 1.3 million tokens per task, against Fable's 21 turns and 130,000 tokens. On the long terminal tasks, it was Fable that spiraled — up to 64 turns and 1.5 million tokens, sometimes straight into a timeout. The pattern flips by task shape: K3 grinds steadily on bounded work, Fable can spiral on open-ended long-horizon work.

How to use it

Name your own task families the way this benchmark did: something bounded and code-shaped, and something longer and open-ended. Run both models on a handful of real examples from each family before deciding anything permanent.

Track two things per task, not more: how many turns it took to finish, and whether it finished at all. Those two numbers are what separated a model that grinds through work from a model that spirals in this benchmark.

Send the bounded, code-shaped work to whichever model finished in fewer turns without spiraling. Send anything open-ended and long-horizon through the same test before trusting it to run unsupervised.

Where it pays most

A benchmark like this is a map of where each model's failure mode shows up, not a verdict on which model is better. K3's failure mode is cost and effort on tasks it is not suited for. Fable's failure mode, in this benchmark, was spiraling on long open-ended work. Knowing which failure mode you are exposed to on your own workload is worth more than knowing which model scored higher overall.