Top 5 models by benchmark
Each ranking uses the best measured configuration per model family. Spark count and serving runtime can vary across entries.
Decode scaling
Highest aggregate throughput at any measured concurrency using ~256 input / 512 output tokens per request. Higher concurrency can include queueing.
View decode scaling chart- Qwen3.6-35B-A3B NVFP4 (Unsloth)Tool eval completeSafety gate failed
429.21 output tok/s
- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
417.63 output tok/s
- NVIDIA Nemotron 3.5 Bolt 74B-A9BTool eval completeSafety gate failed
234.45 output tok/s
- Thinking Machines Inkling 2 MiniTool eval completeSafety gate failed
229.47 output tok/s
- Qwen3.8 27BTool eval incompleteSafety gate failed
201.49 output tok/s
- Qwen3.6-35B-A3B NVFP4 (Unsloth)Tool eval completeSafety gate failed
429.21 output tok/s
- NVIDIA Nemotron 3.5 Bolt 74B-A9BTool eval completeSafety gate failed
234.45 output tok/s
- Thinking Machines Inkling 2 MiniTool eval completeSafety gate failed
229.47 output tok/s
- Qwen3.8 27BTool eval incompleteSafety gate failed
201.49 output tok/s
- Qwen3.8-Flash-Next NVFP4Tool eval completeSafety gate failed
111.73 output tok/s
- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
417.63 output tok/s
- DeepSeek V4.1 Flash 1017 EXL3-EAQTool eval incompleteSafety gate passed
125.87 output tok/s
- GLM-5.3 FlashTool eval completeSafety gate failed
121.04 output tok/s
- DeepSeek V4 Flash DSparkTool eval completeSafety gate passed
104.82 output tok/s
- DeepSeek V4 Flash Vision-ExpTool eval completeSafety gate failed
79.91 output tok/s
Long-context prefill
Lowest p95 time to first token on the 245,760-token passkey workload at concurrency 1.
View long-context prefill chart- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
98.98 s
- DeepSeek V4.1 Flash 1017 EXL3-EAQTool eval incompleteSafety gate passed
120.40 s
- NVIDIA Nemotron 3.5 Super VL 120B-A12BTool eval completeSafety gate failed
155.41 s
- Thinking Machines Inkling 2 MiniTool eval completeSafety gate failed
155.99 s
- GLM-5.3 FlashTool eval unavailableSafety gate unavailable
163.89 s
- Qwen3.8-Flash-Next NVFP4Tool eval completeSafety gate failed
126.64 s
- NVIDIA Nemotron 3.5 Super VL 120B-A12BTool eval completeSafety gate failed
155.41 s
- Thinking Machines Inkling 2 MiniTool eval completeSafety gate failed
155.99 s
- MiniCPM-V 4.7 35B-A3BTool eval incompleteSafety gate failed
226.64 s
- DeepSeek V4 Flash 0731Tool eval unavailableSafety gate unavailable
259.00 s
- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
98.98 s
- DeepSeek V4.1 Flash 1017 EXL3-EAQTool eval incompleteSafety gate passed
120.40 s
- GLM-5.3 FlashTool eval unavailableSafety gate unavailable
163.89 s
- DeepSeek V4 Flash DSparkTool eval completeSafety gate passed
175.40 s
- DeepSeek V4 Flash Vision-ExpTool eval completeSafety gate failed
185.15 s
Sustained load
Highest aggregate throughput in the 20-minute test at concurrency 8, with zero request errors.
View sustained load chart- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
432.90 output tok/s
- Qwen3.6-35B-A3B NVFP4 (Unsloth)Tool eval completeSafety gate failed
286.66 output tok/s
- Qwen3.8 27BTool eval incompleteSafety gate failed
160.59 output tok/s
- NVIDIA Nemotron 3.5 Bolt 74B-A9BTool eval completeSafety gate failed
145.52 output tok/s
- Thinking Machines Inkling 2 MiniTool eval completeSafety gate failed
145.04 output tok/s
- Qwen3.6-35B-A3B NVFP4 (Unsloth)Tool eval completeSafety gate failed
286.66 output tok/s
- Qwen3.8 27BTool eval incompleteSafety gate failed
160.59 output tok/s
- NVIDIA Nemotron 3.5 Bolt 74B-A9BTool eval completeSafety gate failed
145.52 output tok/s
- Thinking Machines Inkling 2 MiniTool eval completeSafety gate failed
145.04 output tok/s
- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
85.72 output tok/s
- Qwen3.8-Flash-Next NVFP4Tool eval incompleteSafety gate passed
432.90 output tok/s
- DeepSeek V4 Flash DSparkTool eval completeSafety gate passed
111.10 output tok/s
- DeepSeek V4.1 Flash 1017 EXL3-EAQTool eval incompleteSafety gate passed
108.73 output tok/s
- GLM-5.3 FlashTool eval unavailableSafety gate unavailable
84.90 output tok/s
- DeepSeek V4 Flash Vision-ExpTool eval completeSafety gate failed
84.53 output tok/s