Four Models, One Brief
A CEO-model orchestrating disposable frontier-model workers: the first pilot run, measured, not marketed.
The task
Fan an identical brief — write a hermetic unit-test suite for a 296-line pure-logic module (the queue arranger) — to 4 frontier models in parallel, each an agentic worker in its own git worktree. Grade objectively by mutation testing: seed bugs, count how many each suite catches.
Results
| Model | Mutation kill | Tests | Asserts | Cost | Latency |
|---|---|---|---|---|---|
| Kimi K3WINNER | 5/6 | 77 | 136 | $0.65 | 814s |
| Grok 4.5 | 5/6 | 52 | 99 | $0.36 | 163s |
| Qwen3.7 Max | 5/6 | 49 | 71 | $0.11 | 190s |
| Grok 4.3 | 5/6 | 16 | 21 | $0.12 | 102s |
Kimi K3 merged — only suite to cover the side-effecting orchestration layer. Qwen3.7 Max: best coverage-per-dollar. Grok 4.3: most efficient, best for a quick check.
The honest finding
All 4 models tied at 5 of 6 killable mutants (7 seeded, 1 proven behaviourally equivalent by design — an initial-sort flip the greedy re-selection makes irrelevant, so no suite could have killed it) — with an identical blind spot: none pinned an exact score for a non-exempt aged piece.
Test count did not predict bug-catching power — Kimi K3's 77 tests beat Grok 4.3's 16 by breadth, not by a wider kill rate; both killed the same 5. The real differentiator was thoroughness vs. efficiency, not correctness.
Build notes
- Muse Spark 1.1 was excluded — Meta gates it to US developers; we operate from HK.
- Native-Windows quirk: the npm .CMD shim mangled multi-line prompts — fixed by calling the .exe directly.
- The deny-by-default bash allowlist visibly blocked a worker's compound
python && lscommand mid-run — guardrail working.