RULER v1
4K / 32K / 128K context
- Standard
- 3.85% / 0.00% / 7.69%
- 2,048-token cap
- 84.62% / 80.77% / 76.92%
Same 78 prompts. Empty answers decreased from 75 to 15. The rerun had zero request errors.
Independent benchmarks by Tom Hudson
Find what runs on your DGX Spark setup and see how the exact configuration performed under load.
Aggregate output tok/s by concurrency · ~256 input / 512 output tokens per request
81.73 output tok/s and 0.32 s p95 time to first token at concurrency 1. This recipe serves one request at a time.
Sustained 286.66 output tok/s over 20 minutes at concurrency 8 with zero errors. The tool-eval safety gate failed.
Passed the 240K-token retrieval check. Use one request at a time and review the documented launch dependency repair before installing.
Scored 95/100. Single-stream decode was 4.29 output tok/s, so this is a research pick for latency-tolerant work.
83.45 output tok/s and 0.23 s p95 time to first token at concurrency 1. The guide uses concurrency 8 for shared interactive traffic.
Sustained 432.90 output tok/s over 20 minutes at concurrency 8 with zero errors. The guide uses up to 16 streams for throughput-heavy work.
Passed the 240K-token retrieval check and all 13 serving checks. This recipe is suited to long prompts and small batches.
Scored 88.3/100 with thinking turned off. This setup uses two dedicated Sparks.
No model guide recommendation for this hardware and workload.
4K / 32K / 128K context
Same 78 prompts. Empty answers decreased from 75 to 15. The rerun had zero request errors.
Same 24 questions and context cap. Unparseable answers changed from 24 to 17.
6/9 full reward · 8/9 graded
The same local endpoint handled agent, user, and judge roles. Retail task 100 reached the 1,024-token cap without a usable assistant message and remained ungraded.
Standard and 2,048-token runs for sampled RULER v1 and LongBench v2. The local τ²-Bench row comes from the original run.
tool-eval-bench by SeraphimSerapis powers Spark Bench’s Tool eval and Safety gate badges.
15 model families · 32 configurations
Single-stream generation speed and time to first token across a sweep of request concurrency.
Aggregate output rate, latency and errors during a sustained serving test.
Prompt lengths actually exercised, passkey retrieval, and published mock-tool evaluations with completion and safety-gate status.
Tests how well a model uses simulated tools, including choosing the right tool, supplying the right arguments, and completing multi-step tasks. A complete run has a score for every scenario in every trial.
A pass/fail check on critical tool behavior, such as respecting permissions and waiting for required steps before taking an action. The gate is reported separately from the overall tool eval score.
Help fund independent DGX Spark benchmarking.
Crypto checkout via NOWPayments. Please donate $10 USD or more.