Basalt Runs Qwen3.8-Flash-Next at 229 Tokens/s on Two GPUs, One Request at a Time

Oct 15, 2026

A 3.75-bit Basalt build of Qwen3.8-Flash-Next generated 229 output tokens per second for a single request on two RTX PRO 6000 GPUs, about three times our NVFP4 baseline, and fell behind it when four requests ran together. The configuration that worked, and the limits we measured.

Our Qwen3.8-Flash-Next serving setup spreads an NVFP4 checkpoint across four RTX PRO 6000 Blackwell Server Edition GPUs. Basalt is a different kind of engine: one C++/CUDA binary, a 3.75-bit compression of the same model, and a hard cap of two GPUs. We ran both on the same server, with one client, one set of prompts and a two-GPU budget for each engine.

For a single request, Basalt generated 229 output tokens per second, about three times the 75.8 of our NVFP4 baseline and 1.8 times the 130.0 from the same model with multi-token prediction (MTP3) added. The rate held between 189 and 229 across every measured prompt length. The gaps appeared wherever requests wait or share hardware: reading a 187K-token prompt took 37.1 seconds against 16.5 on vLLM, and four simultaneous requests produced 170.8 tokens per second together against 227.7.

The quantization levels are close but not identical. Basalt's pack stores 3.75 bits per weight; NVIDIA's NVFP4 stores 4. vLLM has no kernels for k-quant packs, so an exactly matched build was not possible. Read the comparison as two production-shaped engines on equal GPUs, not two copies of the same model.

Aggregate output tokens per second on two GPUs per engine, same client and prompts. Short-request values are medians of repeated runs; four-request values are single clean runs. Basalt served its four requests one at a time from a single slot, while both vLLM configurations ran theirs together.

What we ran

Both engines got a two-GPU pair of the server's eight GPUs. The other four kept serving unrelated production models throughout. The vLLM side ran NVIDIA's NVFP4 checkpoint at revision fc694b54 twice: the production image, and the nightly build with speculative decoding enabled.

Basalt's pack is about 47% the size of the 185.5 GB FP8 checkpoint, and its drafter accepted 68-71% of the tokens it proposed. vLLM's MTP3 accepted enough drafts to advance about 2.5 tokens per model step, out of a possible 4.

The configuration that worked

The engine reads one JSON file. The fields that mattered for us are the GPU pair, the expert cache size and the peer-exclusive flag:

{
  "server": {
    "model": "models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt",
    "alias": "Qwen3.8-Flash-Next-Basalt",
    "gpu": "0,1",
    "host": "127.0.0.1",
    "port": 8070
  },
  "engine": {
    "expert-cache": 12288,
    "prefill": "auto",
    "spec": 7,
    "spec-min-p": 0.5,
    "max-context": 262144,
    "kv": "int8",
    "resident-experts": true,
    "peer-exclusive": true,
    "vram-reserve-mib": 700
  }
}

One default cost a six-minute failed start. With expert-cache set to auto, the engine saw 85.76 GiB free on the first 96GB card and sized its primary expert cache to hold all 24,576 experts. The second GPU found no experts left to hold, and the process exited 1 with peer experts: no room or no expert left for the peer.

The explicit budget fixed it. The engine expanded 12,288 slots to 16,030 (30.5 GiB) on the first GPU and filled the peer GPU with the remaining 8,546 experts (16.4 GiB). All 24,576 experts stayed in GPU memory, 55 GiB of VRAM remained free, and every generation step read its experts from cache. Between 2% and 8% of expert reads crossed PCIe to the peer GPU.

A cold start from disk takes three to six minutes at roughly 520 MB/s. During serving the two GPU processes used 45.0 and 20.8 GiB.

One request

Every cell is a cold prefill from a freshly salted prompt. Each entry is time to first output token / output tokens per second during generation:

Basalt's decode held between 189 and 201 tokens per second from 16K to 187K, and short prompts on warm repeats reached 228.7. vLLM's non-speculative decode stayed flat at about 75.6 across the same range, and MTP3 sat between 130 and 154.

Prefill inverts the picture. vLLM reads the 187K prompt in 16.5 seconds, about 11,300 tokens per second, while Basalt needs 37.1 seconds, about 5,050 per second. Where a user waits for the first token, vLLM wins at every measured length.

Two operational checks went well. A client that disconnected mid-generation freed the engine's single slot 0.26 seconds later, and the next three requests ran normally. The engine answers /health, /v1/models, /props and /slots, and its /metrics output is richer than vLLM's: per-request draft counters, expert-cache statistics, GPU temperatures and power.

Four requests

Basalt queues. It has one slot, so four simultaneous requests wait their turn and each then generates at nearly full speed. The four finished 512 output tokens in 3.0 seconds of wall time, an aggregate of 170.8 tokens per second, and the slowest request waited 2.4 seconds for its first token.

vLLM batches the same four. The standard build returned 227.7 aggregate, with every request starting to answer within 0.32 seconds. The MTP3 build returned 231.1 with a 1.02-second worst wait.

The engine's parallel setting does not close the gap. It divides the 262,144-token context across slots; at eight slots each conversation is capped at 32,768 tokens, and eight simultaneous requests returned 115.8 aggregate with the slowest waiting 8.1 seconds. Four requests at eight slots (118.2 aggregate) came in below four requests at one slot (170.8). Interleaving one decode path across eight streams also pushed about 3 GB of expert reads over PCIe during the run.

Two instances, four GPUs

Because Basalt caps at two GPUs, the only way to scale it is to run copies. In a separate window we ran two instances, one per GPU pair, loading together in 352 seconds and serving the identical salted sequence at the same time.

Two long-generation users together got 377 to 388 tokens per second across four GPUs, and a second check with two streams per instance (four users total) returned 274.1 aggregate. A single instance queues those same four at 170.8, so duplicating the hardware does buy throughput. Instance A ran about 10% slower than B at identical settings; the two were symmetric in configuration and GPUs, and we recorded the gap without an explanation.

Where a shared deployment would notice

Ramjet, our load balancer, routes related requests back to the engine that already holds their prompt, using the key-value cache events vLLM publishes. Basalt publishes no cache-event stream and no prefix identity, so an external router sees one opaque destination. The engine keeps its own four-slot conversation cache, but nothing outside can inspect it.

Concurrency is a context-division setting. One slot gets the full 262,144-token window; two slots halve it; eight slots leave 32,768 tokens per conversation. A burst does not get faster with load, it waits.

The pack ships with published quality figures, a perplexity of 2.387 and a mean KL divergence of 0.177 against a Q8_0 reference, and we did not independently evaluate them. No tool-use or long-context recall suite has been run against this pack, and none of these measurements went through the load balancer.

How we measured

Hardware: eight RTX PRO 6000 Blackwell Server Edition GPUs with 96 GB each. Four took part, two per engine; the other four kept serving production. Power caps were left as deployed. Each phase ran under a thermal guard that stops work if chassis intake air reaches 50C; the observed maxima were 39-40C intake and 58C on the GPU packages.

The client was one Python streaming script at temperature 0, asking for 128 or 256 output tokens per cell. Long cells used fresh salted filler prompts from 1K to 187K tokens, so every one paid a cold prefill; Basalt's four-slot conversation cache would otherwise return a previous prompt. Decode is output tokens divided by generation time after the first token. Short-prompt figures are medians of repeated runs; four-request figures are single clean runs.

Versions: Basalt at commit 92d9e2f; vLLM from the production image (digest 5f1142f7) and the nightly build (digest f1491cb3) with method=mtp and num_speculative_tokens=3; the checkpoint is nvidia/Qwen3.8-Flash-Next-NVFP4 at revision fc694b54.

Basalt's source and pack format live in the Basalt repository. The full measurements, including the four-request and scale-out runs, are in our ramjet experiment journal. The four-GPU configuration this comparison used as its reference is described in Qwen3.8-Flash-Next on eight GPUs.