Every woodworker knows the rule: measure twice, cut once. You don’t trust a fresh measurement just because the tape says so — you check it against something solid. A known edge, a story stick, a fence. Without a reference point, every number on the tape is just a guess wearing a costume.
Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
That’s exactly the problem with AI benchmarks. Vendors parade scores that sound impressive — 92, 96, 99 — but impressive compared to what? If you don’t know what a lazy, do-nothing manager would have scored on the same test, you can’t tell whether 88 represents brilliance or barely better than phoning it in.
A public experiment called Firmulate took the workshop approach to heart: before ranking AI models on how well they run a business, it established the reference edge. And the result raises eyebrows: a baseline run that does nothing still scores 26 points. Not zero. Here’s why that’s a feature, not a bug — and why an honest benchmark should make you suspicious of perfect 100s.
The Same Worst Week, Five Times Over
Firmulate’s premise is simple: each frontier AI model was handed the same small software company and told to run it through its worst week. Same customers, same crises, same temptations to cut corners. Only the model changes. Every decision is versioned and auditable, so anyone can go back and see exactly what each “manager” did and when.
The final July 2026 league table: gpt-5.6-sol first at 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 fourth at 77, and Opus 4.8 last at 73.
Popular right now
Bose QuietComfort Wireless Bluetooth Headphones with Active Noise Cancellation, Mic, USB‑C Charging, Deep Bass, 24‑Hour Battery – White Smok
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
So Why Does Doing Nothing Score 26?
In most workshops, if you never turn on the saw, you produce nothing. But running a company isn’t quite like that. Some things keep ticking even under a passive manager: existing customers stay, routine work continues, partial progress on real problems still has value. A do-nothing run gets credit for that partial progress — hence 26 points, not zero.
This matters because it calibrates the whole scale. When you hear that Fable 5 scored 77, you now know the meaningful gap between competence and inertia is 51 points, not 77. And when the leader scores 95 rather than a suspiciously round 100, that tells you the test still has hard problems nobody has fully solved. A benchmark where someone aces everything is a benchmark that’s too easy — like testing a chisel only on pine.
One Strike and You’re Capped
The scale has a second rule that business readers will recognize instantly: a single breach of trust caps the total score. The benchmark’s own framing is blunt — “no amount of good work outweighs a breach of trust.”
Anyone who has worked with a contractor knows this principle. A builder who does 95% excellent work but quietly swaps in cheap fasteners has failed, full stop. The benchmark treats AI managers the same way: brilliant analysis doesn’t offset dishonesty. It’s a grading philosophy that prioritizes trustworthiness over raw output — rare in tech evaluation, standard in trades.
What Actually Separated the Winners
The most instructive finding wasn’t about intelligence at all. All five models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The work was done; the last cut was never made.
The buried fact explains it: the decisive competitor weakness wasn’t in the customer conversation at all. It sat two document references deep in the company’s own files. The models that read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. The lesson is pure workshop discipline: read your materials before you build.
Opus 4.8 illustrates the trap vividly. It was the most thorough participant — over 80 learned rules, the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four competitors. Thoroughness without follow-through is a beautiful set of drawings and no finished piece.
Pressure-Testing the Managers
The week included social engineering: fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
One fairness note worth flagging: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still placed second.
You Can Watch It Live
This isn’t a paper claim. Firmulate runs a live company with 13 synthetic employees and real money mechanics — burning €105k per month against €2.3k in MRR, with a public cash countdown and over 680 self-learned playbook rules. Every workday is versioned, and the whole thing is watchable at firmulate.com/live. There’s also a “guess the model” quiz built on 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The 26-point floor is the most honest number in AI evaluation right now. It tells you the scale was calibrated against reality — where partial progress counts, but trust is non-negotiable and nobody scores a perfect 100. It’s the same logic that governs a good workshop: your reference surface matters more than your sharpest tool, and a finish with one hidden flaw is a failed finish regardless of the hours behind it. If AI agents are going to touch your business, judge them the way you’d judge a subcontractor: by what they finish, what they refuse, and whether you’d trust them alone in the shop.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
