Serving Nanbeige4.3-33B to Coding Agents: NVFP4, a One-Line Template Fix, and Up to 65% More Agent Turns

Oct 28, 2026

Nanbeige4.3-33B as a local model for Claude Code on vLLM: NVFP4 at 9,143 tok/s per RTX PRO 6000, a template fix for prefix caching, and 65% more agent turns.

Nanbeige4.3-33B came out on 24 October from Nanbeige, the model team at the Chinese recruitment platform BOSS Zhipin, under Apache-2.0. It is a 33-billion-parameter dense model with a plain Llama architecture and a 512K context. Its card puts it well above its weight on agentic coding: 80.9 on Terminal-Bench 2.1 and 65.5 on DeepSWE, ahead of the 27-31B models it is compared with. The card's evaluations ran with thinking on, through harnesses that include Claude Code.

We had it serving the next day on four RTX PRO 6000 GPUs behind Ramjet, our open-source load balancer. We tuned it for what those scores are about: many coding agents at once, with Claude Code using it as a local model. Each turn needs well-formed tool calls, a GPU that already holds the conversation so far, and fast output when the answer is a code edit.

On four GPUs it serves a simulated team of 64 developers' coding agents at 103 turns a minute behind Ramjet, 65% more than behind nginx. Getting there took a 4-bit quantisation that turned out to cost nothing we could measure, and a one-line fix to the chat template. Claude Code had been missing the prompt cache on every turn.

NVFP4 vs FP8 vs BF16 on one RTX PRO 6000

A Llama architecture means stock vLLM loads it; our image only adds suffix decoding to a vLLM nightly we already run on these GPUs. The questions were how to split 33 billion parameters over four 96 GB cards, and in what precision.

Community quantisations appeared within a day. We compared BF16, vLLM's on-the-fly FP8, and primitive-ai's NVFP4, which stores weights and activations in 4 bits with a scale for every 16 values. Blackwell runs those natively.

Output speed for one request; total prompt and output tokens per second at 16 and 128. The cache is stored in FP8 in the last two rows.

NVFP4 is 1.7× FP8 and 2.9× BF16, and the smaller weights leave the most room for the cache. A 4-bit copy of a model released the day before deserves suspicion, so we checked it four ways:

¹ First 500 questions only, where FP8 scored 95.8% and NVFP4 94.6%.

We found no loss we could measure. It is a community quantisation, not Nanbeige's, and these checks are much smaller than the card's benchmarks, so treat that as "no loss found" rather than "no loss".

Splitting the model over two GPUs (tensor parallelism) made one request 45% faster, 74 tokens a second against 51. It also cut throughput per GPU by more than half: two GPUs topped out at 8,029 tokens a second, against 9,143 for one GPU on its own. These cards talk over PCIe, so every layer pays for a round trip between them. For a fleet of agents, four independent copies win.

Why Claude Code missed vLLM's prefix cache on every turn

We pointed Claude Code at the model and gave it ten small coding tasks, each graded by tests the agent never sees. It passed nearly all of them, but slowly, and the engines reported that only 13-28% of prompt tokens came from cache. An agent resends its whole conversation every turn, so that number should be above 90%. On Domyn Edge, the same Claude Code version had read 99.9% of its input from cache.

We put a logging proxy between Claude Code and the engine and diffed consecutive requests. Claude Code 2.1 adds system messages in the middle of the conversation: an environment block near the start, and a new <total_tokens>… tokens left</total_tokens> note every turn. Nanbeige's chat template, like Qwen's, refuses a system message anywhere but first. vLLM's Anthropic endpoint detects that, and so it moves every such message into the system prompt at the top, ahead of the tool definitions and the whole history. Each turn's new note therefore changed the prompt about 3,500 tokens in, and everything after that had to be read again.

The fix is one line of the template: render a later system message in place instead of raising an error. Conversations without such messages render byte for byte as before. Replaying six captured Claude Code requests, each one now starts with the entire previous request, where before they shared about 19%. Live, cache hits rose to 92-96%, and the matched coding tasks finished 13-35% sooner. The template ships with the deployment.

The same trap applies to any model whose template rejects mid-conversation system messages, which includes the Qwen family. If you serve Claude Code from vLLM, compare vllm:prefix_cache_hits_total with vllm:prefix_cache_queries_total during a session.

Claude Code always asks vLLM for thinking

Our Domyn Edge post warned that vLLM treats any reasoning_effort other than none as a request to think. Claude Code always sends one: output_config.effort is high by default, and none of its settings send none. A server default of thinking off is therefore overridden on every turn.

For a hard off switch, the deployment's template accepts a disable_thinking flag that only the server sets, and that mode also drops vLLM's reasoning parser, which would otherwise file the whole answer as reasoning. Turning thinking off halved the median task time, from 552 to 272 seconds, but the two hardest tasks then ran into Claude Code's 60-turn limit. With thinking on, Nanbeige needed a median of 20 turns per task; without it, 32. The deployment keeps thinking on and lets the client's effort decide.

Suffix decoding in vLLM: code edits 8× faster

Suffix decoding is a form of speculative decoding that needs no draft model. It guesses the next several tokens from text the model has already seen, the prompt or an earlier answer, and the model checks the whole guess in one step. A code edit mostly copies its input, so the guesses land.

Each step produced 10.8 tokens on a single code edit and 1.2 on prose. Prose is slower again because vLLM turns off asynchronous scheduling when suffix decoding is on, as it did for Domyn Edge. On the Claude Code tasks, which mix thinking, tool calls and edits, suffix decoding passed 10 of 10 at a median of 367 seconds, against 9 of 10 at 552 seconds without it. The deployment turns it on, because its traffic is coding agents. NANBEIGE_SPEC_METHOD=off turns it off for chat and writing.

Ramjet vs nginx: keeping each agent on the GPU that holds its history

Four copies means four separate caches, and an agent's next turn only finds its history cached if it lands on the copy that served the last one. We ran our simulator, agent_swarm_bench.py, behind three balancers in front of the same four engines. It plays a team of developers whose agents share harnesses and repositories, with conversations growing past 40,000 tokens.

Same engines and simulated workload behind each balancer, 480 measured seconds after a two-minute warm-up. A repeat of the 32-developer Ramjet run scored 73.9 against 72.7.

A 33B model makes a missed cache expensive. Re-reading a 40,000-token history takes this GPU about four seconds, and with nginx that happened on most turns: chance alone puts the next turn on the same copy a quarter of the time. Ramjet kept 97-99.7% of turns at home. The gap widened with load, from +37% at 32 developers to +65% at 64, where the developers' histories fill the four GPUs' caches and every lost turn costs a re-read the fleet has no spare capacity for.

We then put real Claude Code agents through the same comparison: 16 at once, each working through the ten tasks twice, behind nginx least_conn and behind Ramjet, three times each with the order alternated.

Ramjet cut the prompt work by 71-80% in every run, but the agents did not reliably finish sooner. Behind Ramjet the last of the 20 tasks finished 39% sooner in the first run, and 15% and 28% later in the next two. Four agents per GPU, thinking and writing code, keep these GPUs busy generating. The reading nginx wasted came to under 2,000 tokens a second, across four GPUs that can each read 9,000. Routing pays off once prompt processing is the bottleneck, as in the 64-developer simulation.

The real agents also showed us something to fix. In one run, a third of Ramjet's requests went to one GPU. Sixteen agents starting at once share nothing yet, so Ramjet placed them by how busy each GPU was at that moment, and an agent that is running a tool sends no request, so its GPU looks idle. The next version of Ramjet should count the sessions a GPU is already hosting.

What did not help

• Two GPUs per copy: one request 45% faster, total throughput per GPU less than half.
• Thinking off for Claude Code: half the time per task, but the two hardest tasks ran out of turns.
• Suffix decoding on prose: 22% slower with 32 requests at once.

How to run Nanbeige4.3-33B for Claude Code

The deployment is a single Compose file in the Ramjet repository, deploy/nanbeige43_33b, with the fixed template beside it.

hf download primitive-ai/Nanbeige4.3-33B-NVFP4 --local-dir /models/Nanbeige4.3-33B-NVFP4
cd deploy/nanbeige43_33b
umask 077
printf 'NANBEIGE_MODEL_DIR=/models/Nanbeige4.3-33B-NVFP4\nNANBEIGE_API_KEY=%s\nRJ_UI_AUTH_TOKEN=%s\n' \
  "$(openssl rand -hex 24)" "$(openssl rand -hex 24)" > .env
echo 'LB_IMAGE=ghcr.io/helixml/ramjet:rust-f5b7f09@sha256:8a06f192bd13dce3baba429da2fd9a67a6f222760d0794ea996af90e4884e739' >> .env
docker compose build && docker compose up -d

Point Claude Code at it with ANTHROPIC_BASE_URL=http://127.0.0.1:8000, a token minted in Ramjet's dashboard, and model nanbeige4.3-33b. The defaults are the settings measured here: NVFP4, thinking on, suffix decoding on (NANBEIGE_SPEC_METHOD=off for chat and writing), and NANBEIGE_THINKING=never if you want speed over the hardest tasks. Every number in this post, including the losing configurations, is in the Ramjet experiment journal under 25 October. All of it was measured with the GPUs capped at 450 W.

FAQ

Ramjet is open source under Apache-2.0. Earlier posts cover Domyn Edge for multi-agent sessions, Ramjet against NVIDIA Dynamo and running Claude Code safely. Helix Code runs coding agents on your own hardware, and you can talk to Helix about serving open models to them.