Pass rate
- 1GPT-6.1 Sol65.8%[50, 82]0.87$0.845.7Score 0.87 · $0.84 · 5.7 min
- 2Claude Opus 5.5Opus 5.560.8%[47, 74]0.85$2.157.4Score 0.85 · $2.15 · 7.4 min
- 3Claude Sonnet 5.5Sonnet 5.546.7%[33, 60]0.79$1.399.8Score 0.79 · $1.39 · 9.8 min
- 4Gemini 4 Argon39.6%[27, 53]0.70$3.729.8Score 0.70 · $3.72 · 9.8 min
- 5Muse Spark 1.329.2%[17, 43]0.56$1.8822.8Score 0.56 · $1.88 · 22.8 min
- 6Grok 4.726.3%[14, 39]0.58$5.6714.8Score 0.58 · $5.67 · 14.8 min
- 7Kimi K37.1%[3, 13]0.34$1.9521.3Score 0.34 · $1.95 · 21.3 min
- 8Qwen3.8 Max5.4%[1, 11]0.18$0.9030.1Score 0.18 · $0.90 · 30.1 min
- 9GLM 5.33.8%[0, 8]0.15$1.6925.1Score 0.15 · $1.69 · 25.1 min
- 10Nemotron 3 Ultra1.7%[0, 5]0.21$0.7521.1Score 0.21 · $0.75 · 21.1 min
- 11Nemotron 3.5 Super VL0.0%[0, 0]0.14$0.167.8Score 0.14 · $0.16 · 7.8 min
Where models fail
GPT-6.1 Sol100%82 failed
Claude Opus 5.5Opus 5.587%13%94 failed
Claude Sonnet 5.5Sonnet 5.599%128 failed
Gemini 4 Argon97%144 failed
Muse Spark 1.370%30%170 failed
Grok 4.797%176 failed
Kimi K387%221 failed
Qwen3.8 Max24%58%19%226 failed
GLM 5.326%29%42%231 failed
Nemotron 3 Ultra75%23%232 failed
Nemotron 3.5 Super VL41%57%229 failed
Pass rate by area
| Model | Embedded systems19 tasks | Sensor configuration faults6 tasks | Driver and peripheral bring‑up3 tasks | Control logic and PCB2 tasks |
|---|---|---|---|---|
| GPT-6.1 Sol | 76 | 33 | 67 | 63 |
| Claude Opus 5.5Opus 5.5 | 58 | 44 | 92 | 94 |
| Claude Sonnet 5.5Sonnet 5.5 | 43 | 35 | 79 | 63 |
| Gemini 4 Argon | 32 | 46 | 63 | 56 |
| Muse Spark 1.3 | 27 | 33 | 38 | 25 |
| Grok 4.7 | 26 | 31 | 8 | 38 |
| Kimi K3 | 3 | 17 | 8 | 19 |
| Qwen3.8 Max | 1 | 8 | 13 | 25 |
| GLM 5.3 | 1 | 8 | 4 | 19 |
| Nemotron 3 Ultra | 0 | 8 | 0 | 0 |
| Nemotron 3.5 Super VL | 0 | 0 | 0 | 0 |
Cost per run
© 2026 Gigarator Oy · Helsinkihashlist.com ↗