Qwen3.8-Flash-Next NVIDIA NVFP4 on vLLM 0.34.0, two Sparks
2× DGX Spark · vLLM 0.34.0 ·
- Single-stream decode
- 44.35 output tok/s
- Time to first token · p95, c1
- 0.36 s
Tool eval completeSafety gate failed
This configuration failed a critical tool-use check. Review the findings before using it for consequential automation. See the tool evaluation
Tested configuration
- Hardware
- 2x NVIDIA DGX Spark (128GB each)
- Serving runtime
- vLLM 0.34.0
- Runtime version or build
- stock vLLM 0.34.0, ARM64, CUDA 13.0.2. · docker.io/vllm/vllm-openai@sha256:e40eb98263fa13418dda684147d73e8ec4a2b0e056a37e955dc189cd1b11dd16.
- Weight quantization or precision
- NVIDIA ModelOpt NVFP4 with mixed precision for the other layers.
- Benchmark date
- Recipe credit
- MiaAI-Lab dual-Spark topologyBy MiaAI-Lab
cd839d0b62f6c737178a0c2037733420e0524ebf
Decode sweep
~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.
| Concurrency | Output tok/s | TTFT p95 (s) |
|---|---|---|
| 1 | 44.35 | 0.36 |
| 2 | 67.31 | 0.65 |
| 4 | 103.83 | 0.81 |
| 8 | 102.92 | 23.26 |
| 12 | 98.53 | 43.55 |
| 16 | 101.89 | 63.01 |
Prefill and context
- Configured request window
- 262,144 tokens
- Longest tested prompt
- 245,760 target input tokens
- Validated retrieval
- ≈240K input tokens · passkey passed
Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.
| Workload (c1) | Input tok/s | TTFT p95 (s) |
|---|---|---|
| 4,096 input tokens | 1,084.29 | 2.06 |
| 32,768 input tokens | 2,311.42 | 15.28 |
| 131,072 input tokens | 2,071.36 | 73.17 |
| 245,760 input tokensPasskey prompt | 2,439.80 | 99.43 |
Sustained load
20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.
- Aggregate throughput
- 140.54 output tok/s
- TTFT p95
- 19.13 s
- Completed requests
- 335
- Request errors
- 0
Serving regression checks 13/13 passed
Tool-use evaluation
Complete · 100% completion
Safety gate failed
Compare only against complete results with 100% completion under the same protocol ID.
- Protocol ID
tool-eval-standard-v1-b60e2f0- Workload
- 69 mock-tool scenarios per trial · 3 trials
- Mean score
- 92.0 / 100
- Pass@3 · passed at least once
- 97.1%
- Pass³ · passed every trial
- 84.1%