Qwen3.8-Flash-Next on Eight GPUs: What Actually Helped

Aug 27, 2026

We ran Qwen's Qwen4 preview on two TP4 vLLM engines, rejected the tempting knobs, and found the biggest serving win in routing.

Qwen3.8-Flash-Next landed with the kind of spec sheet that makes inference engineers cancel their afternoon: a 125B-parameter main model with 6B active per token, plus a 51B n-gram embedding table and a 4B MTP module, native 262K context, multimodality, and a preview of the architecture Qwen is exploring for Qwen4.

The official Qwen release and vLLM's day-zero recipe gave us a strong starting point. We wanted to answer a more practical question: what does it take to serve the official FP8 checkpoint well on one eight-GPU RTX PRO 6000 server?

We ended up with two independent TP4+expert-parallel vLLM engines behind ramjet, our prefix-aware load balancer. Then we changed one thing at a time.

Conceptual, not tensor-by-tensor. Architecture labels come from the official FP8 model card and technical report.

The unusual part is not just the MoE sparsity. The 125B main model activates roughly 6B parameters per token, while a separate 51B bigram/trigram table injects short-range memory at layer two. The 48-layer core repeats three Gated DeltaNet layers followed by one Qwen Sparse Attention layer twelve times, with 10 of 512 routed experts plus one separate shared expert active. A separate MTP module proposes future tokens for speculative decoding.

The hardware shape

The node has eight RTX PRO 6000 Blackwell Server Edition GPUs: 768 GB of aggregate GDDR7 ECC capacity and roughly 12.8 TB/s of aggregate local-memory bandwidth. That bandwidth is the sum of each card's local memory—not a claim about GPU-to-GPU links. Each card is configurable up to 600 W, giving the box a 4.8 kW GPU nameplate ceiling. Thermals are a serving constraint, not a footnote.

The official FP8 checkpoint is 172.78 GiB. A four-GPU replica gives it 384 GB of physical VRAM for weights, KV cache, activations and CUDA graphs; two replicas consume all eight GPUs but keep separate processes and separate caches. The vLLM recipe supports TP4 where memory allows, but this exact two-replica RTX topology is our measured deployment, not an upstream reference design. Hardware specifications come from NVIDIA's product page and 8-GPU reference architecture.

This is the same practical idea behind the Helix Sovereign Server deployment guide: keep model weights, prompts, tool calls and agent traces on infrastructure you control, then make the serving layer observable enough to operate safely.

What worked

First, the basic topology worked. The pair passed tool calling, reasoning, multimodal, 199K deep-context and near-native-limit 251K-token tests. A c64 cell completed 128/128 requests at 3,340.5 aggregate output tok/s, split exactly across both engines.

The useful tuning results were less obvious:

That last result was the big one in our controlled affinity/load conflict test. The old router kept charging a long request for its full prefill reservation while it decoded. An idle engine therefore looked cheaper than the busy engine holding the request's prefix cache. Once ramjet shrank the reservation when semantic output began, the returning request stayed on the warm engine. Blocker throughput retained 99.1%, and a separate c32 crossover improved aggregate throughput by 2.7%.

The largest gain did not come from a more aggressive kernel flag. It came from teaching the router when the expensive part of a request had actually ended.

The next comparison: GLM-5.3 Flash

GLM-5.3 Flash arrived with a similar efficiency-first pitch but a larger compute envelope: Z.ai reports 320B total parameters and 18B active per token. Qwen reports a 125B main model, 51B of n-gram embeddings and a 4B MTP module—about 180B across those components—with 6B main parameters active per token.

These are overlapping benchmark names from the vendors' own launch evaluations, not a controlled head-to-head. DeepSWE alone used different sampling settings and context limits; the NL2Repo guardrails and context limits also differ. The comparison is useful directionally, but the deltas are not our measurements. Sources: Z.ai's GLM-5.3 Flash launch post, official GLM model card, and official Qwen model card.

Qwen's model card also reports a wider exact-model profile beyond those four overlapping rows:

Source: Qwen's model card. These benchmarks use different harnesses and task definitions, so the bars describe a profile rather than a single comparable scale. DeepSWE reports the better of two harnesses, SWE-bench Pro uses Qwen's refined task set, and CoWorkBench is Qwen's in-house benchmark.

Our own correctness gate was deliberately narrower: tool calls, reasoning envelopes, multimodal input, long context and exact client/server token reconciliation. Passing it meant “safe enough to continue serving experiments,” not “independently proved intelligent.”

What did not work

More batching sounded promising. Raising max-num-batched-tokens from 8,192 to 16,384 changed median cold TTFT by almost nothing, while mean TTFT regressed 29% and p95 regressed 88%. We rolled it back.

More cache hits were not automatically better either. Lowering ramjet's load penalty increased native prefix hits from 35.9% to 47.9%, but returning-request TTFT got 15.1% worse and useful blocker throughput fell 7.3%. We kept the original balance.

The vLLM recipe UI's generated Simple CPU offload preset was a non-starter on this host: it asked for roughly 220 GiB per rank, or 880 GiB per TP4 engine, on a machine with 125 GiB of RAM. The separate n-gram embedding offload also needed more safe host headroom than the two-engine stack left available. Both stayed on GPU.

The bugs were useful

The first boot at 85% GPU memory failed safely with no space left for KV. At 90%, vLLM's automatic profile offered 38.32 GiB on one boot and only 4.48 GiB on an otherwise equivalent warm boot. We stopped treating auto-sizing as authority and pinned the first valid KV allocation explicitly.

The OpenAI-compatible cached_tokens field also stayed at zero while vLLM's native counters showed real prefix hits. Our harness now reconciles against engine counters instead of trusting that response field with this pinned preview runtime/model combination.

And our rollback code caught us once: it restored the Compose file but omitted the immutable ramjet image override, briefly starting an older incompatible binary. The engines never changed, the load balancer failed closed, and later paths pin the image on every forward and rollback command. We also collected the usual harness scars—one doubled /v1, one wrong base path, and a vision test whose output cap ended inside the model's reasoning.

Where we left it

The qualified configuration uses the official FP8 weights, fixed on-device KV, MTP3, QSA index reuse, an 8,192-token scheduler batch cap, and phase-aware prefix routing. Candidate-specific projected-load scoring is implemented but stayed off because we did not qualify it.

When chassis intake rose above our conservative admission threshold, we stopped the Qwen campaign instead of forcing another benchmark window. The model is paused while the box moves to its next experiment. That is part of the result too: a good serving setup includes a clear point at which you stop tuning.

The full Compose deployment, experiment journal, and router source are public.

Measured on an 8× RTX PRO 6000 Blackwell Server Edition server using Qwen/Qwen3.8-Flash-Next-FP8. Two NUMA-local TP4+expert-parallel vLLM engines were served behind ramjet. These are bounded serving measurements, not general model-quality claims.