Spark Bench

MiniCPM-V 4.7 35B-A3B EXL3 6.08bpw on ExLlamaV3, one Spark, thinking off

1× DGX Spark · ExLlamaV3 ·

Single-stream decode
47.74 output tok/s
Time to first token · p95, c1
2.74 s
Tool eval incompleteSafety gate failed

This configuration failed a critical tool-use check. Review the findings before using it for consequential automation. See the tool evaluation

Tested configuration

Hardware
1x NVIDIA DGX Spark (128GB)
Serving runtime
ExLlamaV3
Runtime version or build
r0b0tlab's ExLlamaV3 fork (branch minicpm-v4.7-dspark @ 5ddae7d12448, ExLlamaV3 v1.5.0 base with sm_121 fixes and the MiniCPM-V 4.7 text architecture), built from source into a host-native venv with torch 2.14.0+cu130 and the host's CUDA 13 toolkit
Weight quantization or precision
EXL3 6.08bpw
Benchmark date
Recipe credit
r0b0tlab MiniCPM-V 4.7 EXL3 GB10 recipeBy r0b0tlabb8859be8b257ddb7602f68ce7b85a7f0fd03f7b4

Decode sweep

~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.

ConcurrencyOutput tok/sTTFT p95 (s)
147.742.74
252.4211.40
452.0731.15
852.5669.86
1251.66110.22
1652.74145.74

Prefill and context

Configured request window
262,144 tokens
Longest tested prompt
245,760 target input tokens
Validated retrieval
≈240K input tokens · passkey passed

Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.

Workload (c1)Input tok/sTTFT p95 (s)
4,096 input tokens1,450.521.61
32,768 input tokens2,344.8713.22
131,072 input tokens1,567.2082.15
245,760 input tokensPasskey prompt1,082.46226.64

Sustained load

20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.

Aggregate throughput
51.35 output tok/s
TTFT p95
72.65 s
Completed requests
128
Request errors
0

Serving regression checks 13/13 passed

Tool-use evaluation

Incomplete pilot · 98.6% completion Safety gate failed

Comparisons require complete results with 100% completion under the same protocol ID. This run is retained as individual evidence.

Protocol ID
tool-eval-standard-v1-b60e2f0
Workload
69 mock-tool scenarios per trial · 3 trials
Mean score
86.0 / 100
Pass@3 · passed at least once
79.4%
Pass³ · passed every trial
79.4%

Read the tool-eval report and scenario findings

Install model

Copy this prompt and paste it into an AI assistant with access to your DGX Spark.