GLM-5.3-Flash on RTX PRO 6000, Part 1: Getting Day-Zero Serving to Work

Aug 27, 2026

We got the 321B-parameter GLM-5.3-Flash serving in NVFP4 on four RTX PRO 6000 GPUs. Here is what worked, what broke, and why it is still an experiment.

We had barely finished testing Qwen3.8-Flash-Next on our eight-GPU server when Z.ai released GLM-5.3-Flash. On paper, it is exactly the sort of model we want for a sovereign agent platform: roughly 321 billion parameters, 18 billion active for each token, native tool use, image and video input, and a declared context window of more than one million tokens.

The obvious question was whether we could serve it well on the same 8× RTX PRO 6000 Blackwell Server Edition node.

The short answer is: we got it running, and the first results are promising, but the day-zero serving stack is not ready for production on this hardware. One NUMA-local four-GPU SGLang engine served the community NVFP4 checkpoint at 63.7 output tokens per second for one request. EAGLE speculative decoding raised that short single-request result to 82.8 tok/s, but did not improve aggregate throughput at eight concurrent requests. Text and a simple tool call worked. A stricter agent protocol suite passed four of five cases. Long-context, multimodal and official-FP8 comparisons are still open.

This is Part 1: what it took to make GLM-5.3-Flash serve at all on RTX PRO 6000, what we learned from the first bounded tests, and which attractive-looking options we deliberately left off. Part 2 will follow once the main vLLM support PR and SGLang support PR settle, the blocking bugs are fixed, and we can build clean upstream-derived images for a fair FP8-versus-FP4 comparison.

Why this model is different

GLM-5.3-Flash is not a conventional transformer with one uniform key-value cache. Its 45 language layers combine 34 KDA linear-attention layers with 11 sparse MLA layers. It has 288 routed experts, selects eight of them for each token, and adds a shared expert. A separate multi-token prediction layer can propose several future tokens for the main model to verify together.

The architecture is efficient because most tokens do not activate the full model and most layers do not repeatedly attend over the entire prompt. The serving consequence is that the runtime has to manage several different kinds of state: paged attention KV, sparse-attention indexer state, KDA recurrent state, speculative-decoding state and CUDA graphs. A flag named “prefix caching” does not magically make all of those states interchangeable or reusable.

That distinction mattered throughout this experiment. The biggest memory surprise was not the model weights. It was the separate KDA state pool.

Measured per GPU in the working TP4 engine. Components are rounded native SGLang startup figures; they explain the allocation rather than claiming byte-exact additivity.

Each rank held 44.97 GiB of weights, 20.80 GiB of FP8 KV and roughly 18.7 GiB of KDA/Mamba state. That state pool exposed 543 slots even though our safe scheduler limit allowed only four requests to run at once. This is why our next memory experiment will cap the KDA pool explicitly before reducing the GPU allocation fraction or enabling another feature. “Free KV memory” is not the only capacity that governs this model.

The checkpoint that fit

The official FP8 checkpoint is about 305.8 GiB. It is a natural quality reference and should fit across all eight GPUs, but one TP8 engine would cross the server's NUMA boundary and create only one scheduler and one cache domain.

For the first feasibility test we chose LibertAIDAI/GLM-5.3-Flash-NVFP4, a 181.3 GiB community conversion made with NVIDIA ModelOpt. It quantizes the large routed-expert feed-forward weights while leaving attention, KDA, shared experts, routers, embeddings, the vision tower and other sensitive parts in BF16. That mixed checkpoint is a better fit for one four-GPU group, letting us eventually consider two independent TP4 engines behind ramjet.

It is important to be precise about what we proved. We loaded and tested one TP4/EP4 engine on GPUs 4–7, close to its CPU and memory socket. The container used 48.88 GiB of host RAM and the machine never fell below 77.25 GiB available during that startup. This proves that one engine fits the node's 128 GB RAM. It does not prove that two engines can load simultaneously. Our two-engine plan starts them sequentially and treats the second loader peak as a separate admission gate.

The 181 GiB checkpoint also does not require 181 GiB of CPU RAM. The loader can stream shards into GPU allocations rather than holding the whole model in ordinary memory. In fact, the first Hugging Face Xet download stalled after 4.6 GiB while consuming about 5.5 GiB of RAM. Disabling Xet and using four ordinary HTTP workers sustained roughly 80–110 MiB/s at about 200 MiB of RAM. The boring downloader won.

“Blackwell supported” was not specific enough

Our GPUs report compute capability SM120. The launch recipes covered datacenter Blackwell parts such as B200, GB200, B300 and GB300, but not RTX PRO 6000. That difference was not cosmetic.

The dedicated vLLM image could not select a valid sparse-MLA cache and kernel path for GLM's rope-free attention geometry on SM120. Both FP8 and BF16 attempts failed before serving. The blocker is tracked in vLLM issue #53963; a separate cache-slot bounds bug made that path an especially poor day-zero comparison candidate. We stopped there rather than turning a broken loader into a benchmark.

SGLang got farther, but the generic Blackwell defaults still did not describe this card. Community FP8 work on RTX PRO 6000 had to disable a DeepGEMM optimization that expects newer tensor-memory instructions, switch the sparse-attention path to BF16 KV with TileLang, and reduce a decode pipeline stage so its dynamic shared-memory requirement fit SM120.

The NVFP4 conversions had another problem: the loader classified some BF16 KDA tensors as expert-packed FP4 tensors and allocated the wrong shape. The exact compatibility build we tested fixed that name matching, preserved the ModelOpt metadata used by the draft path, and sliced activation scales correctly for expert-parallel ranks.

That build worked, but it came from a small third-party compatibility repository that did not carry a detected license and replaced six SGLang source files. We hash-verified the files and pinned the image for internal feasibility testing. We will not redistribute it or call it a production runtime. A successful launch is evidence for the upstream fixes, not permission to skip provenance.

The configuration that actually served

Our conservative baseline was:

The engine became healthy in 434 seconds. Weight loading took 213 seconds; scheduler and tokenizer readiness followed at 323 and 337 seconds. This is not a model that belongs behind a casual 60-second readiness timeout.

The launch was clean: no restart, OOM, CUDA error, NCCL error or NVIDIA Xid. It reserved just over three million FP8 KV tokens per rank and left 8.16 GiB of GPU memory per rank after startup. The API exposed the expected model through the same conventional authenticated /v1 prefix used by OpenAI-compatible clients. We kept the earlier model-named Caddy paths only as compatibility aliases.

What the first measurements say

We kept the live tests short because the server is air-cooled and the point was configuration discovery, not a heroic saturation graph. Each cell ran once under an intake-air guard, capped output length, reconciled client token usage against SGLang's native counters, and stopped on errors or thermal limits.

With MTP disabled, one engine produced 63.7 aggregate output tok/s at c1, 183.8 at c8, 229.3 at c16 and 239.2 at c24. The scheduler was intentionally capped at four running requests, so the higher-concurrency cells include queueing. They describe the safe baseline, not the final capacity of four GPUs or the eight-GPU node.

Adaptive EAGLE speculation used up to five prediction steps, top-k one and six draft tokens. It raised c1 output throughput by 30%, from 63.7 to 82.8 tok/s, and cut that short request's time to first token from 529.5 ms to 154.3 ms. At c8, aggregate throughput moved backward by 0.9%, from 183.8 to 182.1 tok/s, while p95 time to first token became 9.9% worse.

It also made startup more expensive. Scheduler readiness grew from 256 to 347 seconds, peak allocation rose by about 1.6 GiB per GPU, and KDA state capacity fell from 543 to 234 slots. The draft path did gain a larger main KV pool plus its own 2.40 GiB allocation, but that is not automatically the best trade for concurrent agents.

So MTP stays off in the checked-in recipe. The low-load gain is real enough to investigate, but a single-session win is not a universal serving win. Part 2 will compare a smaller fixed EAGLE 3/1/4 graph with the adaptive profile against a scheduler-matched MTP-off control.

Correct responses were not the same as correct agents

The deterministic text test passed. A simple typed addition tool passed. All client and native token counts matched. Then we ran a five-case agent protocol corpus at concurrency one.

Four cases passed: ordinary text, two parallel required streamed tool calls, an automatically selected streamed tool, and a reasoning-plus-tool-result history. The fifth response had the correct tool-call envelope and finish reason, but violated the JSON schema inside its arguments. A required nullable note field was neither null nor the requested value. An isolated deterministic replay failed the same way.

That is a 4/5 protocol result, not a pass. It could be the quantized checkpoint, the model, the chat template, the parser or the experimental runtime. Until we compare against official FP8 with upstream code, we do not know which. We stopped the sequence there, so this article makes no long-context, multimodal or end-to-end quality claim.

There was another correctness warning before we even reached long prompts: the checkpoint did not supply FP8 KV scaling factors, so SGLang used 1.0. Short text can look perfectly healthy while accumulated cache error damages recall later. We will not qualify FP8 KV without bounded 32K and 128K recall tests.

The million-token context is not a serving result

The model declares a 1,048,576-token context. We served it with a 262,144-token cap.

That was deliberate. An open SGLang CUDA-graph issue reports an abort on the first decode token after a cold prefill above roughly 262K, while eager mode works. Disabling graphs merely to make a million-token prompt return would create a different performance profile and hide the defect. Context support advances when recall, cancellation, latency, concurrency and cache accounting pass together—not when the allocator accepts a big number.

The same caution applies to prefix caching. The architecture has several state groups, and day-zero vLLM reports observed zero automatic-prefix-cache hits. SGLang recommends leaving prefix caching enabled, but did not publish a repeated-prefix GLM result for this hardware. Ramjet can still make approximate prefix-affinity decisions, but we will not credit it with a cache-speedup until native counters show real reusable state.

Things we deliberately did not turn on

SGLang's cookbook includes HiCache, a 32 GB host-memory tier, and a Mooncake-backed L3 tier. vLLM also exposes NIXL prefill/decode transfer, Mooncake transport and MooncakeStore shared caching. These are interesting systems, but they are not one generic “make cache faster” switch.

GLM's transferable state includes KDA recurrence and sparse-attention/indexer data as well as ordinary KV. Each runtime defines its own grouping, ownership, hashing and telemetry. On our 128 GB host, a 32 GB cache tier would also consume loader safety margin we have not yet proved we can spare. First we need two sequentially loaded GPU-resident engines and a measured local-prefix hit. External cache comes later.

We also left all-reduce fusion, data-parallel attention, two-batch overlap and alternative SM120 MoE kernels alone. Some target other Blackwell variants; some do not compose with EAGLE; some are unfinished. A day-zero stack has enough moving parts without combining five unqualified optimizations into one unexplained result.

The operational gotchas mattered too

Several problems had nothing to do with matrix multiplication:

• A disk-space alert referred to the root and Docker filesystem, not the roomy ZFS model pool. Cleaning stopped legacy containers, superseded images and build cache recovered about 140 GiB on root without deleting the current Qwen or GLM snapshots.
• Our original thermal policy required 40°C intake air before starting a test. The chassis idled at 42–43°C even with every GPU at zero utilization, so two guarded attempts correctly ran no workload. We changed admission to 46°C while retaining a 50°C abort threshold. The successful baseline peaked at 42°C intake and 65°C GPU temperature.
• The engine did not stop gracefully within 60 seconds. Docker eventually killed it with exit 137, but OOMKilled=false, and all GPU and host memory was reclaimed. Exit 137 was a shutdown defect, not evidence of an OOM.
• After a test, the containers were intentionally stopped and the public API was therefore down. Recovery had to start the engine, wait several minutes for health, then publish Ramjet and test the full authenticated Caddy path. A healthy GPU node is not the same thing as a healthy inference API.
• One recovery probe timed out because real client requests arrived first and queued ahead of it. Those clients returned HTTP 200. The guard still failed closed instead of declaring ambiguous health.

These details are why the deployment recipe contains guarded smoke tests, native token reconciliation, immutable model and image checks, API recovery, and a conventional /v1 ingress—not only an engine command.

Where we left it

The best configuration we can defend today is still an experiment: one SGLang NVFP4 TP4/EP4 engine, 262K served context, four running requests, 90% static allocation, FP8 KV, FlashInfer sparse MLA, Triton linear attention, CUTLASS MoE and MTP off. It is running behind Ramjet and authenticated Caddy, but the runtime's third-party source status, the 4/5 tool result, untested long-context recall and open upstream defects prevent production promotion.

The next safe steps are small: cap the oversized KDA state pool, raise the scheduler and graph batch together from four to eight, compare fixed and adaptive EAGLE, run salted prefix-reuse probes, and test 32K/128K recall. Only then do we start the second TP4 engine and measure whether two independent cache domains beat one TP8 engine for concurrent agent sessions.

Part 2 will begin from new immutable upstream-derived images after the vLLM and SGLang support work merges and the relevant SM120, ModelOpt, MTP-pool and long-context bugs are fixed. That article will test official FP8 against NVFP4, one TP8 engine against two TP4 engines, real prefix reuse through Ramjet, and the longer agent/multimodal correctness suite. If NIXL prefill/decode transfer or Mooncake still makes sense after the local cache is understood, we will test those as separate architectures rather than bolt them onto the winning graph.

The full GLM-5.3-Flash deployment study, experiment journal, and qualified experimental recipe are public.

Measured on one NUMA-local group of four RTX PRO 6000 Blackwell Server Edition GPUs inside our eight-GPU node, using the community LibertAIDAI/GLM-5.3-Flash-NVFP4 checkpoint and a pinned experimental SGLang compatibility build. Results are short serving measurements from this machine and workload, not model-quality claims or an endorsement of the unlicensed runtime source.