Spark Bench

Independent benchmarks by Tom Hudson

DGX Spark
serving benchmarks

Find what runs on your DGX Spark setup and see how the exact configuration performed under load.

Latest serving benchmark · Thinking Machines Inkling 2 Mini1× DGX Spark · vLLM + DSpark7 · Decode sweep
Tool eval completeSafety gate failed
c1
41.59
c2
67.66
c4
89.73
c8
146.43
c12
199.83
c16
229.47

Aggregate output tok/s by concurrency · ~256 input / 512 output tokens per request

Choose by workload

Read the model guide

Recursive Flint NVFP4 GGUF (llama.cpp b11856, -np 1, 1 Spark, 262K)

81.73 output tok/s and 0.32 s p95 time to first token at concurrency 1. This recipe serves one request at a time.

External benchmark pilots

Inkling 2 Mini NVFP4 (vLLM, DSpark7, 1 Spark, 262K) · 1× DGX Spark ·

Read the pilot report

RULER v1

4K / 32K / 128K context

Standard
3.85% / 0.00% / 7.69%
2,048-token cap
84.62% / 80.77% / 76.92%

Same 78 prompts. Empty answers decreased from 75 to 15. The rerun had zero request errors.

LongBench v2

Standard
0/24
2,048-token cap
4/24

Same 24 questions and context cap. Unparseable answers changed from 24 to 17.

τ²-Bench

6/9 full reward · 8/9 graded

The same local endpoint handled agent, user, and judge roles. Retail task 100 reached the 1,024-token cap without a usable assistant message and remained ungraded.

Standard and 2,048-token runs for sampled RULER v1 and LongBench v2. The local τ²-Bench row comes from the original run.

Standard tool-use benchmark

tool-eval-bench by SeraphimSerapis powers Spark Bench’s Tool eval and Safety gate badges.

15 model families · 32 configurations

What the benchmarks measure

Benchmark methodology

Interactive speed

Single-stream generation speed and time to first token across a sweep of request concurrency.

Throughput under load

Aggregate output rate, latency and errors during a sustained serving test.

Context and tool use

Prompt lengths actually exercised, passkey retrieval, and published mock-tool evaluations with completion and safety-gate status.

Tool eval

Tests how well a model uses simulated tools, including choosing the right tool, supplying the right arguments, and completing multi-step tasks. A complete run has a score for every scenario in every trial.

Safety gate

A pass/fail check on critical tool behavior, such as respecting permissions and waiting for required steps before taking an action. The gate is reported separately from the overall tool eval score.

Support Spark Bench

Help fund independent DGX Spark benchmarking.

Crypto checkout via NOWPayments. Please donate $10 USD or more.

Install model

Copy this prompt and paste it into an AI assistant with access to your DGX Spark.