SGLang vs DwarfStar vs vLLM+DSpark: Running DeepSeek 4 on the RTX Pro 6000

Aug 14, 2026

Production benchmarks from our Helix agent fleet on 8× RTX PRO 6000: the engine journey from DwarfStar (122 tok/s) to SGLang (653 tok/s) to vLLM+DSpark (946 tok/s box aggregate, 204 tok/s single-stream). Same hardware, 7.8× throughput.

We run a fleet of coding agents on an 8× RTX PRO 6000 Blackwell workstation. Each card has 96 GB of GDDR7. The model is DeepSeek-V4-Flash-0731 — the refresh behind the headline 89% ARC-AGI-1 / 61.4% ARC-AGI-2 numbers — served at FP8 (~156 GB) with DeepSeek's DSpark speculative decoding.

Over the last two weeks we worked through three inference engines — DwarfStar (ds4), SGLang with the 0xSero SM120 patch, and vLLM with SparkInfer — and production now runs vLLM + DSpark. Here's the journey, the numbers, and why we landed where we did.

The hardware: RTX PRO 6000 Blackwell

Before the benchmarks, it's worth being precise about what's in the box. An RTX PRO 6000 is not a rack-mount data-centre accelerator. It's a dual-slot blower card that drops into a standard workstation chassis — the same category as the RTX 6000 Ada before it, but built on the Blackwell architecture (GB202, SM120) with double the memory:

Our box is a Threadripper 7980X workstation with eight of these cards — 768 GB of GDDR7 in a single air-cooled tower that plugs into a wall outlet. That's the whole point: H100-class VRAM without the rack, the PDUs, or the liquid cooling.

Where it sits across the board

The closest comparisons people ask about are DGX Spark, RTX 6000 Ada, and H100. Here's how the RTX PRO 6000 stacks up:

A few observations from actually running on these:

• vs RTX 6000 Ada (48 GB, ~$6,800): double the VRAM, ~1.9× the memory bandwidth, and Blackwell's FP4/FP8 tensor cores. The 96 GB is the difference between fitting DeepSeek-V4-Flash Q4 (~164 GB) on two cards vs needing four.
• vs DGX Spark (128 GB unified LPDDR5x, ~$4,000): the Spark has more raw capacity but a fraction of the bandwidth (273 GB/s vs 1.8 TB/s) — better suited to prototyping than serving a large MoE model at fleet concurrency.
• vs H100 (80 GB HBM3, $25-30K+ per module): the H100 has more bandwidth (3.35 TB/s) and NVLink, but the RTX PRO 6000 has 20% more VRAM, sips 300 W less, needs no liquid cooling, and costs roughly a third of an H100 system.

The pricing matters for the fleet story. Eight RTX PRO 6000s — 768 GB of GDDR7 — cost roughly what two H100 SXM modules do on the street. That's the arithmetic that makes a self-hosted agent fleet on this class of hardware viable.

What we found next is that the engine you put on top of these cards determines far more of the real-world throughput than the card itself.

Why the engine matters for MoE

DeepSeek-V4-Flash is a mixture-of-experts model: 256 experts, 8 active per token. The FP8 checkpoint is ~156 GB. Inference engines handle MoE very differently:

• Pipeline parallelism (DwarfStar's approach) splits layer groups across GPUs. Each token serializes through N stages, so only ~2 of 8 GPUs are active at any instant. Decode throughput is capped by the pipeline depth.
• Tensor parallelism (SGLang/vLLM's approach) shards every layer across all GPUs. All GPUs work on every token. Decode uses 100% of the available compute — but the communication overhead is higher.
• Speculative decoding (DSpark) — a small draft head proposes multiple tokens per step; the big model verifies them in a single forward pass. When acceptance is high, decode jumps several-fold.

The RTX PRO 6000's 1.8 TB/s GDDR7 bandwidth and 96 GB per card give plenty of headroom on paper. The question is whether the engine can actually keep the GPUs fed.

The engine journey, in one table

Same eight cards all the way through. The software changed; the silicon didn't.

DwarfStar (ds4): where we started

DwarfStar (dwarfstar.sh, commit 84cc882) is a narrow C/CUDA inference engine from antirez, specialised for DeepSeek V4 Flash. It served the model with pipeline parallelism across 4 stages (paired GPUs), continuous batching, compressed KV (~9.5 KiB/token), and disk-persisted KV cache keyed by rendered prompt prefix — so agent restarts skip re-prefill and get ~7× better warm TTFT.

Topology

All 8 GPUs, arranged as 4 pipeline stages with TP pairs:

Device list: --gpu-devices 0,2,4,6,1,3,5,7 (all homes first, then partners).

DwarfStar benchmarks (Q4-imatrix, 8 GPUs)

Peak aggregate: 122 tok/s at ~32 concurrent. Beyond that, requests queue gracefully — 48 concurrent held at 118.5 tok/s with zero errors, all 200s.

The ceiling comes from pipeline-parallel decode: each token serialises through 4 PP stages, so only ~2 of 8 GPUs are active per micro-step. GPU utilisation under load: 8-25% (homes ~24%, partners ~8%). The hardware isn't the bottleneck — the PP topology is.

Single-stream decode was a respectable 46.6 tok/s on the grouped Q4 kernels. DwarfStar also has a DSpark speculative decode path, but it's gated off when native session batching is active — the two are mutually exclusive at the pinned commit.

The killer feature DwarfStar still has: disk-persisted KV cache. Agent sessions send full conversations every request. DwarfStar caches the rendered prompt prefix to disk, so turn-2+ TTFT is ~7× faster than a cold re-prefill. SGLang's RadixAttention caches in GPU memory only — survive a restart and the cache is gone.

SGLang + 0xSero SM120 patch

The 0xSero/deepseek-v4-flash-sm120 project builds a CUDA extension for the missing FlashMLA sparse-decode kernel on SM120 (Blackwell), injected via PYTHONPATH hook into an unmodified lmsysorg/sglang:deepseek-v4-blackwell image. SGLang runs pure tensor parallelism (TP=4) — no pipeline serialisation, so all GPUs work on every token.

SGLang benchmarks (FP8, 4 GPUs, TP=4, EAGLE)

Peak aggregate: 349 tok/s at 16 concurrent, 324 tok/s at 48-way. GPU utilisation under load: 88-97% — all four GPUs saturated, doing useful work every cycle.

The CUDA graph collapse (and fix)

The first run with max-running-requests 32 hit a nasty cliff: 349 tok/s at 16-way, then 78 tok/s at 32-way. Root cause: CUDA graphs were captured to batch size 32, but EAGLE with num-draft-tokens 2 doubles the verification batch. ≤16 running requests stayed on-graph (16×2=32). At 17+, requests fell off the CUDA graph path to eager mode, and throughput cratered.

Fix: max-running-requests 16. Excess requests queue instead of running off-graph. With that change, 32-way recovered to 237 tok/s and 48-way hit 324 tok/s — both on-graph, all GPUs saturated.

2× TP=4 topology: 653 tok/s on 8 GPUs

Once SGLang worked, we deployed two TP=4 instances across all 8 GPUs — one on GPUs 4-7, one on GPUs 0-3, fronted by a conversation-affinity sticky load balancer (radix-cache locality):

653 tok/s aggregate at 64-way, split 50/49 across the two instances — near-linear scaling. Compared to DwarfStar's original 122 tok/s peak on the same 8 GPUs: 5.3× improvement, using the same hardware.

vLLM + DSpark: the current production stack

We'd originally written vLLM off — the dspark method failed to launch on 8-card Blackwell (vllm#49418), and one community report showed DSpark lowering saturated throughput (vllm#49369). Then the jacklarmer/deepseek-v4-flash-0731-sm120 recipe landed with SparkInfer's B12X kernel set, and the 0731 refresh shipped with the DSpark draft head bundled in the checkpoint (MXFP4 experts native in the FP8 file). We came back to vLLM.

Topology: two TP=4 instances behind a sticky LB

Helix ── Caddy :80 /deepseek-v4-flash/* (bearer auth)
    └─ ds4-metrics-proxy (sticky LB, failover, shims, Prometheus)
        ├─ dspark-0731   GPUs 0-3
        └─ dspark-0731-b GPUs 4-7

• Engine: vLLM + SparkInfer/B12X (cu132 build), env-driven in-image launcher. DCP_SIZE=1, KV dtype fp8_ds_mla, block size 256, VLLM_USE_B12X_FP8_GEMM=0.
• Model: deepseek-ai/DeepSeek-V4-Flash-0731 FP8 (~156 GB, DSpark draft head bundled) — expert_dtype resolved to 'fp4' at boot.
• Driver: upgraded 580 → 595.84 (2026-08-11) for the Blackwell branch.

vLLM + DSpark benchmarks

946.6 tok/s aggregate at 32-way across the whole box — versus DwarfStar's 122 tok/s on the same eight cards. That's a 7.8× improvement. And the single-stream numbers are the real eye-opener: DSpark's speculative decoding takes decode from 39-46 tok/s up to ~204 tok/s — roughly 5× on its own.

Gotchas we hit (each one cost real time)

1. Context accounting: engines run MAX_MODEL_LEN=393216 but the proxy advertises 262144. Zed/Helix pace compaction on the advertised window using client-side token estimates that undercount the rendered prompt — vLLM hard-rejects at the true limit and the turn dies. And a thread already over the engine limit can't even run compaction. The engine ceiling must exceed the advertised window or wedged threads never self-heal.
2. API key + probes: VLLM_API_KEY must equal the Caddy bearer (clients' Authorization passes through). The proxy's health probes need DS4_UPSTREAM_TOKEN or vLLM 401s them and the LB marks a healthy engine down (serving keeps working — the tiles lie).
3. Env is launch-time: docker compose up needs the right env per service; recreating engine containers requires re-connecting the docker network aliases (dspark/dspark-b). A plain docker restart keeps env + aliases; recreation does not.
4. Recipe blockers (all pre-armed in our compose): KV dtype must be fp8_ds_mla (nvfp4 asserts on SM120), DCP_SIZE=1, num_speculative_tokens≥5 fixed by the DSpark block, block size 256.

The proxy shims

The harness-compatibility shims live in the proxy (engine-agnostic): strip max_tokens/max_completion_tokens ≥100k, flatten content-parts arrays (incl. Zed's empty {"type":"text"} part), drop invalid reasoning_effort ("none"). vLLM is more tolerant than SGLang was, but the shims stay — they're what make harness switches non-events.

Head-to-head comparison

The headline: 7.8× aggregate throughput and ~4-5× single-stream decode over where we started, on identical hardware. DSpark's speculative decoding is the single biggest lever for our fleet — agent workloads sit at the favourable end of DSpark acceptance (community: ~73% on code@0, ~33% on prose; our ~3.3 tok/step confirms it).

Production decision (2026-08-11)

Production now runs vLLM + DSpark on both GPU halves behind the sticky-load-balancing proxy. Rollback paths are warm:

1. SGLang stack intact — launch.sh (A: GPUs 4-7, B: GPUs 0-3) + repoint proxy DS4_UPSTREAM. Pre-0731 checkpoint still on disk.
2. Driver — 580 modules kept for both kernels; apt-get install nvidia-driver-580-open + reboot.
3. DwarfStar — GGUF weights deleted (would need re-download); config preserved.

When to pick each engine

Pick vLLM (with a DSpark-capable recipe) when

• Decode throughput at fleet concurrency is your goal — DSpark's ~5× single-stream jump compounds across every agent session
• You have TP-friendly GPU counts (4 or 8)
• You can run the recipe's kernel constraints (SM120, correct KV dtype, driver ≥590)
• You need the broad ecosystem (multi-LoRA, structured output, request priority)

Pick SGLang when

• You need the 0xSero-style CUDA-graph tuning and RadixAttention prefix sharing
• DSpark/SparkInfer isn't an option on your stack yet
• You're on older drivers where vLLM's DSpark path doesn't launch

Pick DwarfStar when

• You need disk-persisted KV cache for fast agent restart TTFT
• Single-stream decode without speculative decoding is your baseline
• You want a narrow, specialised engine with no moving parts

The bottom line

The RTX PRO 6000 is capable hardware, but the engine determines what you actually get out of it. On the same 8 GPUs, with the same class of model, we measured a 7.8× difference in aggregate throughput between our first engine (DwarfStar, 122 tok/s) and our current one (vLLM+DSpark, 946.6 tok/s) — and a ~10× jump in single-stream decode (19 → 204 tok/s) once speculative decoding entered the picture. That's not a GPU gap — that's a software architecture gap.

DwarfStar's pipeline parallelism was the right call for the L40S era where its 120 tok/s benchmarks were published. Blackwell is fast enough that PP serialisation dominates, and fast enough to feed a full-time speculative decoder. The recipe that unlocks it — SparkInfer's B12X kernels plus a bundled DSpark draft head — is moving fast; we're tracking r-image releases because each one carries meaningful fixes.

We started this post calling it "SGLang vs DwarfStar vs vLLM." Two weeks later the honest title is: try them all, on your own box, with your own workload. On ours, vLLM + DSpark won by a mile.

Production benchmarks from an 8× RTX PRO 6000 Blackwell workstation (AMD Threadripper 7980X, 256 GB RAM, driver 595.84). Model: DeepSeek-V4-Flash-0731 FP8 (~156 GB, DSpark draft head bundled, MXFP4 experts). vLLM + SparkInfer/B12X via a cu132 build, two TP=4 instances behind a sticky-LB proxy. Historical: DwarfStar commit 84cc882 (Q4-imatrix GGUF); SGLang via 0xSero SM120 patch (EAGLE, CUDA graphs bs=32). Full configs and benchmark scripts in the Helix GitHub.