Local AI performance numbers are easy to misunderstand. A model might be described as running at 250 or even 300 tokens per second, while another deployment of “the same model” produces fewer than 50. The usual explanation is not one magic setting. Model checkpoint, numerical precision, inference engine, speculative decoding, request concurrency, and the benchmark itself all affect the result.
We wanted a practical answer, so we deployed the optimized Qwen3.8-27B stack on a single NVIDIA RTX PRO 6000 Blackwell GPU and then removed one optimization at a time. The result was clear: the high-speed claim is real, but DFlash2 speculative decoding was the largest part of the gain—not simply switching from vLLM to SGLang.
The System We Tested
The machine used for this deployment has:
- One NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
- 96GB-class GPU memory (97,887 MiB reported)
- A 300-watt GPU power limit
- Ubuntu 24.04 and Docker
- A native 262,144-token model context ceiling
The optimized production stack runs:
- Target model:
RadixArk/Qwen3.8-27B-NVFP4 - Draft model:
z-lab/Qwen3.8-27B-DFlash2 - Inference server: SGLang
- KV cache: FP8 E4M3
- Concurrent request limit: 8
- User interface: Open WebUI connected through an OpenAI-compatible API
The restored production service reports a 951,552-token shared cache pool. That is enough for one request to use the model’s full 262K context, or for multiple requests to share the available cache.
The Four Configurations
To separate the optimizations, we tested four deployments on the same machine:
- SGLang + NVFP4 + DFlash2 — the optimized production configuration.
- SGLang + NVFP4 without DFlash2 — isolates speculative decoding.
- vLLM + the same NVFP4 checkpoint — isolates the inference backend.
- SGLang + the official FP8 checkpoint without DFlash2 — measures the effect of NVFP4 versus FP8.
Every run used the same client, prompt order, output limits, temperature zero, and fixed random seed. We warmed each service before timing it so startup compilation was not counted as generation time.
The Results
The fixed-length test forced exactly 3,072 output tokens across three technical prompts. It is the cleanest raw single-stream comparison:
- SGLang + NVFP4 + DFlash2: 161.7 tokens/second
- SGLang + NVFP4 without DFlash2: 71.5 tokens/second
- vLLM + NVFP4: 66.0 tokens/second
- SGLang + FP8: 48.5 tokens/second
We then ran 32 official GSM8K reasoning questions. Single-user results were:
- SGLang + NVFP4 + DFlash2: 226.1 tokens/second, 31/32 correct
- SGLang + NVFP4 without DFlash2: 70.1 tokens/second, 30/32 correct
- vLLM + NVFP4: 64.1 tokens/second, 31/32 correct
- SGLang + FP8: 47.8 tokens/second, 31/32 correct
At eight simultaneous requests, aggregate throughput reached:
- SGLang + NVFP4 + DFlash2: 720.0 tokens/second, 32/32 correct
- SGLang + NVFP4 without DFlash2: 365.1 tokens/second, 31/32 correct
- vLLM + NVFP4: 308.3 tokens/second, 31/32 correct
- SGLang + FP8: 212.0 tokens/second, 31/32 correct
These are end-to-end completion-token rates, including prompt processing and HTTP overhead—not an isolated kernel microbenchmark.
What NVFP4 Actually Changed
NVFP4 reduces key model operations from 8-bit to 4-bit precision. On Blackwell GPUs, this matters for two reasons: less data has to move through memory, and the GPU has hardware designed to process supported 4-bit operations at higher throughput.
In the matched SGLang tests, changing FP8 to NVFP4 improved single-user speed by approximately 47%. Under the eight-user reasoning load, the aggregate improvement reached 72%.
NVFP4 also reduced the model’s reported weight footprint in SGLang from 28.47GB for FP8 to 21.68GB for the NVFP4 target. That extra room can be used for cache, concurrency, or a speculative draft model.
This is a real hardware and precision advantage—but it does not explain a jump from roughly 48 to 226 tokens per second by itself.
How Much Did SGLang Matter?
With the exact same NVFP4 checkpoint and no speculative decoder, SGLang produced 70.1 tokens per second on the single-user reasoning test. vLLM produced 64.1.
That made SGLang about 9% faster for one reasoning request and about 18% faster in the eight-user aggregate test. Those are useful gains, especially in production, but they are much smaller than the total improvement.
The backend also changed memory allocation. vLLM reported a larger 1.87-million-token KV pool, while matched SGLang reported roughly 1.08 million. That does not mean one allocator is universally better: CUDA graph reservations, hybrid-attention state, cache layout, and scheduling policies differ between engines.
The practical lesson is that the inference server matters, but it should be tested with the same checkpoint and workload. Comparing unrelated saved benchmark runs can easily produce the wrong conclusion.
DFlash2 Was the Biggest Accelerator
Autoregressive language models normally generate one token at a time. DFlash2 adds a smaller draft model that proposes a block of possible future tokens. The full 27B target then verifies those candidates in parallel and accepts the valid sequence.
During our reasoning benchmark, SGLang commonly accepted around four to six tokens from an eight-token proposal block, with some intervals above seven. That allows the system to advance several tokens per target-model verification step.
Adding DFlash2 to the same SGLang/NVFP4 target produced:
- A 2.26× improvement in the fixed-length test
- A 3.23× improvement in single-user reasoning
- A 1.97× improvement in eight-user aggregate throughput
The benefit varies by workload because draft-token acceptance depends on how predictable the output is. Structured reasoning can be highly favorable; unusual prose, code, or rapidly changing topics may accept fewer draft tokens.
Is 300 Tokens per Second Real?
Yes—but it needs context.
We measured 226.1 tokens per second averaged across the complete 32-question single-user reasoning run. Inside the server, individual decode intervals exceeded 300 tokens per second. The fixed-length text test averaged 161.7, while eight simultaneous reasoning requests reached 720 aggregate tokens per second.
So “300 tokens per second” is possible on this hardware and stack, but it is not a universal per-request guarantee. Prompt length, output style, DFlash acceptance, concurrency, cache state, and whether the measurement excludes request overhead all change the number.
A credible performance claim should always identify whether it means:
- One user’s sustained generation rate
- The fastest observed interval
- Aggregate throughput across concurrent users
- Decode-only speed or full end-to-end latency
What Is the Quality Tradeoff?
The small GSM8K check did not show an obvious collapse in mathematical accuracy. The four configurations scored between 30/32 and 32/32.
However, 32 questions are a smoke test—not proof of identical model quality. They do not measure coding, instruction following, factual recall, safety behavior, multilingual performance, or long-context reliability.
The generated reasoning was also not byte-for-byte identical across backends and precisions, even at temperature zero. Quantized kernels, GPU scheduling, batching, and floating-point tie-breaking can change the greedy path while still producing the same final answer.
There is another important distinction: the previous baseline service used a separate uncensored FP8 fine-tune, while the optimized production target is the RadixArk NVFP4 checkpoint with a compatible DFlash2 drafter. They belong to the same Qwen3.8-27B family, but they are not behaviorally identical checkpoints. Refusal style and other fine-tuned behavior should be evaluated separately.
Finally, both serving engines warned that the FP8 KV cache did not include calibrated scaling factors and therefore defaulted to 1.0. Our small accuracy sample remained strong, but a broader evaluation is appropriate when answer fidelity is more important than maximum context capacity.
What We Run in Production
After completing every control test, we restored and verified the optimized deployment:
- SGLang serving Qwen3.8-27B in NVFP4
- DFlash2 active with an eight-token proposal block
- Full 262,144-token context support
- Eight concurrent requests
- OpenAI-compatible API for application integration
- Open WebUI for browser-based access
- Health checks and an actual completion verified after restoration
The original vLLM container remains preserved and stopped, making rollback straightforward.
The Practical Takeaway
The benchmark demonstrates why production LLM optimization has to be treated as a complete system problem.
On this Blackwell workstation, NVFP4 provided a meaningful 4-bit hardware advantage. SGLang added another modest backend improvement. DFlash2 then delivered the largest acceleration by reducing the number of expensive sequential target-model steps.
The combined system produced more than 3.3× the fixed-text speed and 4.7× the single-user reasoning throughput of the FP8 SGLang control. Just as importantly, the testing showed where the headline number applies—and where it does not.
If your organization is deciding between cloud inference and an on-premise GPU deployment, the correct question is not simply, “How fast is this model?” It is: “How fast is this exact model, precision, serving engine, speculative decoder, context configuration, and workload?”
That is the level at which reliable AI infrastructure decisions are made.
NexGen Compute designs and optimizes local AI workstations, inference servers, and production model deployments. If you want to evaluate the right GPU, model format, and serving stack for your workload, talk to an AI infrastructure expert.
