RULER v1
86.92% at 4K · 86.15% at 32K · 86.15% at 128K
Macro scores across 78 cases. All eight needle-retrieval variants scored 100% at 128K.
2× DGX Spark · TensorFold ·
This run has ungraded attempts. Its score reflects only part of the tool evaluation. See the tool evaluation
be3ed857ab1055ca47b36134d22691f00f2bb94a~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.
| Concurrency | Output tok/s | TTFT p95 (s) |
|---|---|---|
| 1 | 83.45 | 0.23 |
| 2 | 129.24 | 0.43 |
| 4 | 195.56 | 0.68 |
| 8 | 282.69 | 1.02 |
| 12 | 362.59 | 1.44 |
| 16 | 417.63 | 2.11 |
Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.
| Workload (c1) | Input tok/s | TTFT p95 (s) |
|---|---|---|
| 4,096 input tokens | 2,010.45 | 1.51 |
| 32,768 input tokens | 2,673.36 | 11.24 |
| 131,072 input tokens | 2,614.74 | 48.97 |
| 245,760 input tokensPasskey prompt | 2,436.01 | 98.98 |
20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.
Serving regression checks 13/13 passed
Comparisons require complete results with 100% completion under the same protocol ID. This run is retained as individual evidence.
tool-eval-standard-v1-b60e2f086.92% at 4K · 86.15% at 32K · 86.15% at 128K
Macro scores across 78 cases. All eight needle-retrieval variants scored 100% at 128K.
9/24 correct (37.5%)
Sampled six domains. Ten responses hit the 128-token output cap before yielding a parseable answer.
5/9 tasks earned full reward
Airline 2/3, retail 2/3, telecom 1/3. Two telecom tasks hit the step limit on the retry.
RULER and LongBench used sampled workloads. The custom τ²-Bench pilot used local Qwen as the agent, simulated user, and natural-language judge.
tool-eval-bench by SeraphimSerapis powers Spark Bench’s Tool eval and Safety gate badges.