Pass rate
- 1GPT-6.1 Sol64.6%[49, 82]0.87$0.845.7Score 0.87 · $0.84 · 5.7 min
- 2Claude Opus 5.5Opus 5.559.2%[46, 72]0.84$2.157.4Score 0.84 · $2.15 · 7.4 min
- 3Claude Sonnet 5.5Sonnet 5.545.4%[32, 59]0.79$1.399.8Score 0.79 · $1.39 · 9.8 min
- 4Muse Spark 1.329.2%[17, 42]0.56$1.8822.8Score 0.56 · $1.88 · 22.8 min
- 5Kimi K36.7%[2, 13]0.33$1.9521.3Score 0.33 · $1.95 · 21.3 min
- 6Qwen3.8 Max4.2%[0, 9]0.17$0.9030.1Score 0.17 · $0.90 · 30.1 min
- 7GLM 5.33.3%[0, 8]0.14$1.6925.1Score 0.14 · $1.69 · 25.1 min
- 8Nemotron 3 Ultra1.7%[0, 5]0.21$0.7521.1Score 0.21 · $0.75 · 21.1 min
Where models fail
GPT-6.1 Sol100%85 failed
Claude Opus 5.5Opus 5.588%12%98 failed
Claude Sonnet 5.5Sonnet 5.599%131 failed
Muse Spark 1.370%30%170 failed
Kimi K387%222 failed
Qwen3.8 Max25%57%18%229 failed
GLM 5.326%29%41%232 failed
Nemotron 3 Ultra75%23%232 failed
Pass rate by area
| Model | Embedded systems19 tasks | Sensor configuration faults6 tasks | Driver and peripheral bring‑up3 tasks | Control logic and PCB2 tasks |
|---|---|---|---|---|
| GPT-6.1 Sol | 76 | 33 | 54 | 62 |
| Claude Opus 5.5Opus 5.5 | 58 | 44 | 75 | 94 |
| Claude Sonnet 5.5Sonnet 5.5 | 43 | 35 | 67 | 62 |
| Muse Spark 1.3 | 27 | 33 | 38 | 25 |
| Kimi K3 | 3 | 17 | 4 | 19 |
| Qwen3.8 Max | 1 | 8 | 0 | 25 |
| GLM 5.3 | 1 | 8 | 0 | 19 |
| Nemotron 3 Ultra | 0 | 8 | 0 | 0 |
Cost per run
© 2026 Gigarator Oy · Helsinkihashlist.com ↗