Blocked by the regression gate
8 passed2 regressed1 still failing1 not run
Pass rate · v8
72.7%
Down 19.0% vs v7
Regressions
2
block promotion
Judge score
0.81
Down 5.1% vs v7
Cost per case
$0.0042
Up 7.7% vs v7
| version | Pass rate | Judge score |
|---|---|---|
| v3 | 58% | 66% |
| v4 | 67% | 71% |
| v5 | 75% | 77% |
| v6 | 83% | 82% |
| v7 | 92% | 86% |
| v8 | 73% | 81% |
The refund case fails on every model, so it's the prompt.
| Rank | Run | ||||
|---|---|---|---|---|---|
| 1 | 91.7% (best) | 0.86 (best) | 1.88 s | $0.0039 | |
| 2 | 83.3% | 0.82 | 2.04 s | —not measured | |
| 3 | 75.0% | 0.78 | 640 ms (best) | $0.0006 (best) | |
| 4 | 72.7% | 0.81 | 1.96 s | $0.0042 |
No intervals were supplied, so small gaps between runs may not be real.