Running GLM-5.3-Flash on 2, 4 or 8 RTX PRO 6000 GPUs

Sep 14, 2026

Our measured GLM-5.3-Flash TP2 recipe reaches 388 output tok/s and nearly 6,000 prefill tok/s on two RTX PRO 6000 GPUs, then scales by adding independent replicas.

One copy of GLM-5.3-Flash now runs across two RTX PRO 6000 Blackwell Server Edition GPUs at 388.2 aggregate output tokens per second with four simultaneous requests. The same tensor-parallel two-GPU unit can be repeated once, twice or four times on a 2, 4 or 8 GPU server.

We currently run two of those GLM replicas on GPUs 4-7 of a Helix Sovereign Server. Qwen3.8-Flash-Next keeps GPUs 0-3. Ramjet exposes both models through one OpenAI-compatible endpoint and sends each request only to an engine that owns the requested model.

This is the production follow-up to our day-zero GLM-5.3-Flash experiment. The model moved from one fragile four-GPU canary that failed an agent protocol case to two healthy TP2 replicas with a corrected tool parser, measured prompt processing, and a clear concurrency limit.

One replica always uses two GPUs

Tensor parallelism splits one model copy across several GPUs. With tensor parallel size two (TP2), each GLM replica owns a pair of 96 GB GPUs, one scheduler and one prompt cache. Adding GPUs adds complete replicas instead of stretching one process across the whole server.

This shape gives a 4 GPU machine two independent queues and an 8 GPU machine four. One replica can restart while the others remain available. Related requests can return to the replica that already holds their reusable prompt data.

A larger TP4 or TP8 process may help a checkpoint that cannot fit on two cards. This quantized checkpoint fits TP2, so larger tensor parallel groups would spend more GPUs on communication and leave fewer independent schedulers. We have not found evidence that this model needs that trade.

The checkpoint that makes TP2 possible

The official FP8 checkpoint is too large for a two-card replica. We use the community ormandj/GLM-5.3-Flash-W4A16-NVFP4-K32-Experts-FP8-WO checkpoint at immutable revision ee0989a944b0e213589191d7fca63af825a0741e.

It stores the routed expert weights in NVIDIA FP4 while retaining selected sensitive components at higher precision. Each GPU rank loaded 81.44 GB of target weights and 3.20 GB of speculative NextN weights. Target key-value cache, draft cache and KDA recurrent state brought the process close to the usable memory limit of each card.

Our engine is a pinned SGLang v0.4.3 SM120 build. The pinned vLLM image we tested could not serve this model on RTX PRO 6000: the sparse MLA configuration has no rotary-position section, while the available packed FP8 path required one. vLLM v0.29 still lacked the required merged SM120 path when we qualified the SGLang setup. We recorded the failed vLLM load instead of publishing a throughput number from a server that never became ready.

The initial SGLang tool parser also encoded JSON null as the string "null" for a nullable argument. Our derived image applies a narrow, hash-checked parser fix. The complete runtime, checkpoint and parser authority is public.

The settings that passed

The model combines sparse attention with KDA linear-attention state. Its memory limit is governed by both token cache and per-request recurrent state, so the usual “raise concurrency until the GPU is full” approach is unsafe.

The core of the engine command is short enough to recognise, but not enough to reproduce the deployment by itself:

command:
  - serve
  - --model-path=/models/glm53-flash-w4a16-e4m3-k32
  - --served-model-name=glm-5.3-flash
  - --tp=2
  - --context-length=524288
  - --max-total-tokens=500000
  - --max-running-requests=4
  - --max-mamba-cache-size=28
  - --chunked-prefill-size=6144
  - --max-prefill-tokens=6144
  - --kv-cache-dtype=fp8_e4m3

The checked-in Compose recipe also pins the sparse-attention, MoE, graph-capture, multimodal and speculative-decoding paths. Copying only the lines above would silently select different kernels.

Each replica gets a distinct GPU pair and compilation cache:

services:
  glm-a:
    deploy: {resources: {reservations: {devices: [{driver: nvidia, device_ids: ["0", "1"], capabilities: [gpu]}]}}}
  glm-b:
    deploy: {resources: {reservations: {devices: [{driver: nvidia, device_ids: ["2", "3"], capabilities: [gpu]}]}}}
  glm-c:
    deploy: {resources: {reservations: {devices: [{driver: nvidia, device_ids: ["4", "5"], capabilities: [gpu]}]}}}
  glm-d:
    deploy: {resources: {reservations: {devices: [{driver: nvidia, device_ids: ["6", "7"], capabilities: [gpu]}]}}}

Keep only glm-a for two GPUs, add glm-b for four, and use all four services for eight. Start them sequentially. A cold replica took about 17 minutes to load weights, tune collectives, capture graphs and run serving warmups on our machine.

Four active requests are the TP2 sweet spot

We measured one TP2 replica with forced 256-token code responses. Every cell reconciled the client usage with SGLang's native request and generated-token counters. No late compilation landed inside a measured interval.

Four requests produce the best aggregate result. At eight, only four can run and the rest wait. Aggregate throughput falls 7.7% from the tuned four-request peak, and the median wait for the first token rises above 1.4 seconds. We therefore kept --max-running-requests=4 instead of consuming the last device-memory margin on more KDA state.

The per-stream number excludes prompt reading and measures answer generation after a stream begins. The aggregate number covers the wall-clock test. That is why the single-request per-stream rate can be slightly higher than the aggregate rate.

Prompt processing reaches nearly 6,000 tok/s

Agents often send much more input than they generate. Repository maps, tool schemas and conversation history all have to be read before the first output token appears.

Increasing the prefill chunk from 4,096 to 6,144 tokens raised cold 8K prompt processing by 7.4%. The larger chunk cost 24,288 shared token slots, or 4.6% of the pool. An 8,192-token chunk failed the engine's own serving warmup, even after we reduced the pool.

These rates include scheduling and first-token overhead. Repeat tests also reconciled the engine's cached-prompt counters, but the table stays on cold prompts so a cache hit cannot make the prefill result look faster than it is.

Capacity on 2, 4 and 8 GPUs

The measured unit is one TP2 replica. Multiplying its four-request result gives a useful planning ceiling for larger machines:

The four-GPU layout is live and both replicas passed direct health, model discovery and routed requests. We have not run a saturated two-replica benchmark because the machine was serving production traffic during qualification. The eight-GPU row is the same Compose shape repeated four times, not a full-node measurement. Host scheduling, PCIe traffic, power and cooling can keep real scaling below the arithmetic result.

This replica layout also scales availability. On four or eight GPUs, a failed engine removes part of the capacity instead of the whole model. Ramjet keeps unhealthy replicas out of rotation and continues routing to the remaining pair or pairs.

Intelligence and serving correctness are different tests

Z.ai reports strong agent and coding results for the official GLM-5.3-Flash model. Three useful examples from its model card are:

Those are Z.ai's results for the model, not measurements from our quantized checkpoint or our hardware. Quantization, chat templates, tool parsers and serving runtimes can all change the behaviour users receive.

Our local gate asks a narrower question: can this exact checkpoint and server preserve the agent protocol? It passed five deterministic cases covering ordinary text, required parallel tool calls, automatic tool selection, and a reasoning-plus-tool-result history. All five responses matched the expected structure after the nullable parser fix. That is a 5/5 serving-correctness result, not a replacement for Terminal-Bench or DeepSWE.

Ramjet routes model names and replicas separately

Our current eight-GPU server is intentionally mixed. One Qwen TP4 engine owns GPUs 0-3. Two GLM TP2 engines own GPUs 4-5 and 6-7. Ramjet returns both model IDs from /v1/models, then restricts each request to replicas that serve its selected model.

Within the GLM pool, the router balances load and keeps related prompts close to reusable cache state. The two GLM engines publish the same model name, so clients do not need to know which pair handles a request. The dashboard still shows the physical topology and token use per model.

The same arrangement works on an all-GLM machine. Configure one, two or four upstreams, give every upstream ownership of glm-5.3-flash, and leave client configuration unchanged as the server grows.

What remains specific to this build

The checkpoint and SGLang runtime are community builds pinned by revision and digest. Our parser layer is currently identified by an exact node-local image ID. Rebuilding or distributing it requires reviewing the upstream source and publishing the derived image under an immutable registry digest.

The model accepts a 524K context in this recipe, but this article reports prompt tests only through 64K. FP8 cache scales defaulted to 1.0 because the checkpoint did not supply calibrated values. Longer recall and multimodal behaviour need their own quality tests.

We also rejected DFlash2 for this deployment. It improved one-request decode by 4.3%, then lost 5.5% at the retained four-request operating point and cut cold prefill by roughly one third. The existing adaptive EAGLE path remains faster for the concurrent agent workload we run.

The raw measurements, rejected configurations and exact deployment files live in the public Ramjet experiment journal and GLM TP2 recipe.

The Helix Sovereign Server has eight RTX PRO 6000 Blackwell Server Edition GPUs and 768 GB of GPU memory. It can run four GLM-5.3-Flash TP2 replicas, or split the GPU pairs between models as we do in production. Talk to Helix about a deployment.