We Doubled Our Inference Throughput by Reading a Log Line

Aug 23, 2026

Our 8× RTX PRO 6000 box looked saturated at 4,256 tok/s. It wasn't — it was queueing behind a state cache we'd never configured. One flag took c192 from 3,833 to 7,882 tok/s, and gating it honestly meant rebuilding our test harness.

We serve Qwen3.8-27B to our coding agent fleet on eight RTX PRO 6000 Blackwell cards — one SGLang engine per GPU, NVFP4 weights, DFlash2 speculative decoding. Our published aggregate ceiling was 4,256 tok/s at 256-way concurrency.

We thought that was saturation. It was a queue.

The log line nobody had read

Every engine had been printing this at boot, for weeks:

max_running_requests is capped to 12 by the mamba state cache
(max_mamba_cache_size=62, 5 state slots per request)

Nothing in our config asked for that. Qwen3.8 is a hybrid model — Gated DeltaNet linear attention in three of every four layers, full attention in the fourth. The linear-attention layers carry long-range information in a fixed-size recurrent state, and that state is allocated per in-flight request from its own pool. Twelve requests per engine. Eight engines. A hard fleet ceiling of 96 concurrent requests, no matter what we sent.

Our load balancer was happily dispatching 256.

We wrote a small sampler that polls what the LB dispatched against what each engine reports actually running (sglang:num_running_reqs) versus waiting (sglang:num_queue_reqs). The picture was unambiguous:

Every engine pinned at exactly 12. Never 13. And c192 was slower than c128 — past the knee we were just accumulating waiting requests. The 4,256 tok/s "ceiling" in our own docs was that same plateau with more people standing in line.

The obvious fix is a trap

SGLang helpfully suggests the remedy in the same log line: raise --max-mamba-cache-size. So we did, on one engine fenced out of production.

It worked, and it was terrible:

The state pool and the KV cache come out of the same static allocation. Sixty-six extra slots ate 24.4 GB of KV — about 369 MB per slot, and at five slots per request that's 1.85 GB of recurrent state per concurrent request. Converted into KV, one extra concurrent request costs ~23,000 tokens of cache. The baseline gives each running request ~29,000 tokens.

In other words: an added slot costs almost exactly as much cache as a slot can use. The shipped 12/349K configuration sits near the natural balance point. Our agents run 18K-token prompts and 200K-token cached sessions — they want more cache per request, not more slots. A 45K-token pool couldn't hold a single one of our real sessions.

That's a dead end dressed up as a fix.

The flag that actually worked

--mamba-ssm-dtype=bfloat16 halves the precision of the recurrent state itself. Same number of bytes buys twice the slots:

Double the concurrency for 1.9% of the cache. After rolling all eight engines two at a time:

The sampler confirmed the mechanism rather than letting us infer it: run_total now peaks at exactly 200, 25 on every engine, and c256 queues 56 — precisely 256 − 200. Time-to-first-token at c128 went from 3.99s to 0.221s, because the wait was the queue, not the work.

For context, our vLLM+DSpark stack on the same box saturates at 7,890.9 tok/s. This SGLang stack now measures 7,882.6 — within 0.1% — while keeping roughly twice the per-stream decode speed. The trade-off we'd documented between the two stacks simply stopped existing in that direction.

Then we had to earn the right to ship it

Here's the uncomfortable part. --mamba-ssm-dtype=bfloat16 is a numerics change. It halves the precision of the exact mechanism the model uses to carry information across long distances. Throughput is not evidence of correctness.

Our agent test harness covered protocol shape: does the model emit well-formed tool calls, are streamed deltas reassembled correctly, does reasoning survive a tool turn. All useful — and all nearly irrelevant to the question "does a lower-precision recurrent state forget things?"

So we extended it along the two axes that actually matter here.

Needle-in-a-haystack, by depth. A new case type builds a filler document of a requested size, plants facts at fractional depths, and asks for them back. Depth matters more than size for this failure mode: if a fixed-size state degrades, it degrades at distance. Our deepest case puts five facts at 1%, 25%, 50%, 75% and 99% of a 199,482-token prompt — 76% of the model's context window.

Real multi-turn sessions. Not a canned transcript: each turn replays the assistant's actual tool calls, feeds back one tool result per call quoting the real tool_call_id, appends a follow-up user message, and validates the next response. Binding results to call IDs is the part of an agent session that genuinely breaks, so we take the IDs from the response rather than inventing them.

We ran both against an unmodified engine and a bf16 engine with an identical salt, so the documents were byte-identical. Result: 79 matched requests across four corpora and two sampling profiles, no regression on any case, turn, or depth. All five needles recalled from 199,482 tokens, on both.

Two bugs in our own tests

Building the harness was more instructive than running it.

Our matcher failed a correct answer. Asked to report two tool results, the model replied Engine A: 27,604 / Engine B: 83,521 — exactly right. Our exact-substring check demanded 27604 and rejected the thousands separator. We very nearly recorded a phantom regression against a change that was fine.

A test that trips on digit grouping is worse than no test. It blocks good work, and once you learn to expect noise from it, it buries the real regression it exists to catch. The matcher now ignores separators only between digits — 27,604 matches 27604, while a,b still doesn't match ab and a different number still fails.

Our expectations attached to the wrong turn. For a case combining a long document with a tool session, the required answer fragments landed on turn 0 — which is a tool call with no prose at all. Every correct run would have failed. We caught that one before it ran, but only because we were checking the built prompts rather than trusting the code.

Both bugs share a shape: the test was wrong in a direction that looks like a finding. That's the failure mode to fear when you're gating a change you already want to ship.

The takeaway

The bottleneck wasn't the GPU, the model, or the KV cache. It was a state pool sized by a default, announced in a log line at every single boot, that nobody had read.

Two things we're taking forward. Instrument the mechanism, not just the outcome — the throughput number told us we'd plateaued, but only the running-versus-queued sampler told us why, and that distinction was the difference between "we're saturated" and "we're 2× off". And before you gate a change, check that your gate can actually see the thing you're worried about. Ours measured protocol conformance and would have cheerfully green-lit a model that had quietly stopped remembering the start of a long conversation.

Measured on an 8× RTX PRO 6000 Blackwell Server Edition server. Model: Qwen3.8-27B NVFP4 (Inferact/Qwen3.8-27B-NVFP4) with the z-lab/Qwen3.8-27B-DFlash2 block-8 draft, one TP=1 SGLang engine per GPU behind our ramjet load balancer. All request-generating runs under an intake-air thermal guard with fresh cache-namespace salts.