RULER v1
4K / 32K / 128K context
- Standard
- 3.85% / 0.00% / 7.69%
- 2,048-token cap
- 84.62% / 80.77% / 76.92%
Same 78 prompts. Empty answers decreased from 75 to 15. The rerun had zero request errors.
1× DGX Spark · vLLM + DSpark7 ·
This configuration failed a critical tool-use check. Review the findings before using it for consequential automation. See the tool evaluation
~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.
| Concurrency | Output tok/s | TTFT p95 (s) |
|---|---|---|
| 1 | 41.59 | 0.28 |
| 2 | 67.66 | 0.61 |
| 4 | 89.73 | 6.86 |
| 8 | 146.43 | 3.13 |
| 12 | 199.83 | 3.38 |
| 16 | 229.47 | 2.85 |
Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.
| Workload (c1) | Input tok/s | TTFT p95 (s) |
|---|---|---|
| 4,096 input tokens | 1,117.46 | 3.01 |
| 32,768 input tokens | 2,363.60 | 12.75 |
| 131,072 input tokens | 2,007.46 | 63.84 |
| 245,760 input tokensPasskey prompt | 1,548.23 | 155.99 |
20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.
Serving regression checks 10/13 passed
Compare only against complete results with 100% completion under the same protocol ID.
tool-eval-standard-v1-b60e2f04K / 32K / 128K context
Same 78 prompts. Empty answers decreased from 75 to 15. The rerun had zero request errors.
Same 24 questions and context cap. Unparseable answers changed from 24 to 17.
6/9 full reward · 8/9 graded
The same local endpoint handled agent, user, and judge roles. Retail task 100 reached the 1,024-token cap without a usable assistant message and remained ungraded.
Standard and 2,048-token runs for sampled RULER v1 and LongBench v2. The local τ²-Bench row comes from the original run.
tool-eval-bench by SeraphimSerapis powers Spark Bench’s Tool eval and Safety gate badges.