Serving Domyn Edge on RTX PRO 6000

Oct 21, 2026

Domyn Edge is a 4B model trained in Europe, with open weights, data and training code. We served it the day after release: FP8 doubled throughput, thinking off made tool calls reliable, suffix decoding made code edits up to 8× faster, and Ramjet served 53-61% more coding-agent turns than nginx on the same four GPUs. It also runs on a gaming RTX 4090 with the full 256K context.

Domyn Edge came out on 20 October, and it is European from end to end. Domyn trained it on Leonardo, the EuroHPC supercomputer at CINECA in Bologna, and published the weights, the 8.4-trillion-token first-stage training data, the training code and fourteen intermediate checkpoints under Apache-2.0. Its strongest languages are English, Italian, Spanish, French and German, and the tokenizer was extended for the last four.

It is a 4-billion-parameter model, and Domyn does not pretend it is a frontier coder: Qwen3.5-4B beats it on most rows of Domyn's own benchmark table, including SWE-bench Verified (40.0 against 33.6). A model whose training data you can audit, trained on European public infrastructure and small enough for one consumer GPU, has few rivals.

We had it serving the day after release. On four RTX PRO 6000 GPUs behind Ramjet, our open-source load balancer, it served 226 coding-agent turns a minute, 53% more than the nginx setup Domyn's own batch tool uses. On a gaming RTX 4090, a single code edit streamed at 927 tokens per second.

Running it before vLLM could

No vLLM release can load Domyn Edge yet. Support sits in an open pull request (vllm-project/vllm#62544), and the model card says to build vLLM from that branch. The pull request turned out to be two new Python files and three registry lines, with no CUDA, so we copy them onto a vLLM nightly we already trust, check their hashes, and skip the compile. The image builds in about four seconds.

The model is cheap to host. Only 12 of its 36 layers look at the whole conversation; the other 24 look back 1,024 tokens. vLLM stores long-term cached attention data for those 12 layers only, so one 96 GB RTX PRO 6000 holds 1.7 million tokens of reusable prompt data, and a 24 GB RTX 4090 holds 277,000.

FP8 doubled the throughput

The first benchmark was slow for a 4B model: 12,900 tokens per second on one GPU with 64 requests in flight, and fewer (12,200) with 256. The GPUs reported 100% utilisation while drawing 278 W of their 600 W.

A profile found the matrix multiplications running on Ampere-generation kernels. That is what cuBLAS picks for BF16 on this Blackwell card. A microbenchmark put its BF16 ceiling at about 270 TFLOPS and its FP8 ceiling at 544. So we stored the weights and the cache in FP8:

That is 2.2× the default. Perplexity on text the model has never seen (Ramjet's own 2026 documentation) rose by 1.2%. These runs used the Base model, which shares the SFT model's architecture.

Thinking off made tool calls reliable

The chat model starts every answer with a thinking block. Through our agent protocol suite, which checks tool arguments, parallel tool calls, multi-turn tool use and recall from documents up to about 128K tokens, thinking hurt. With it on, 80-86% of cases passed on two of three test sets: the model made one tool call where the task needed two, and lost facts in long conversations. With it off, every case passed and tasks used three to five times fewer tokens. Both our deployments start the engine with thinking off.

Our own benchmark then switched it back on. It sends reasoning_effort: low, and vLLM treats any value other than none as "think, please", overriding the server default. The pass rate fell to 57-100% until we sent none. Some agent harnesses send one by default, so check yours. Ramjet passes the field through unchanged.

Suffix decoding: fast for code, slow for prose

A coding agent's answers repeat much of its input: the function it is editing, the file paths, the lines around a change. Suffix decoding exploits that. It guesses the next several tokens from the prompt and from earlier answers, and the model checks the whole guess in one step. We tested it on real excerpts of Ramjet's Rust source with edit instructions, and on 500-word essays in five languages, where there is nothing to copy.

Output tokens per second with suffix decoding (up to 16 guessed tokens per step) against none, same prompts, greedy decoding.

On code edits each step produced 7-9 tokens. On prose it produced about 1.3, barely better than no guessing, and the RTX PRO 6000 still lost up to 31%, more than wasted guesses explain. The engine log names the culprit: "Async scheduling not supported with suffix-based speculative decoding and will be disabled." No setting we tried won that back. Both deployments turn suffix decoding on because our traffic is coding agents, and DOMYN_SPEC_METHOD=off turns it off for chat and writing.

Four GPUs, four caches, one router

A 4B model fits on one GPU with room to spare, so four GPUs means four independent copies. It also means four separate caches, and whether a request finds its prompt already processed depends on which copy it lands on.

Domyn publishes domyn-swarm for batch jobs. It puts nginx with least_conn balancing in front of the model copies, sending each request to whichever copy has the fewest open connections. We kept domyn-swarm's job runner and swapped the balancer. The test batch asked 4,000 questions about 400 documents of about 8,000 tokens each. It took 304 seconds behind nginx and 208 behind Ramjet, which sent each document's questions to the copy that had already read it.

Coding agents gained more. Our simulator (agent_swarm_bench.py) runs a team of developers sharing agent harnesses and repositories, with conversations growing past 40,000 tokens:

Same engines and simulated workload behind each balancer, 480 measured seconds after a two-minute warm-up. The relative rows come from a later run, in which a repeat of marginal scored 204 turns a minute at 96 developers.

The difference is where a conversation's next turn lands. With four copies, chance puts it on the same copy as the previous turn 25% of the time, and nginx managed 34%. Ramjet kept 92% of turns at home. Every turn that moves makes a new copy re-read 40,000-50,000 tokens of history, and that re-reading is where nginx lost its turns and its time.

How Ramjet scores cache overlap mattered too. Its default scoring gave only 5% more turns than nginx, because every conversation starts with the same harness prompt and all four copies looked equally warm. Scoring only the overlap beyond the coldest copy (marginal) gave +31-38%, and scoring against the warmest copy (relative) gave +53-61%. Relative is now the default for this deployment. Three repeat runs of the same setup varied by about 7% either way, so read the chart as a comparison and not a capacity guarantee.

A gaming RTX 4090 against the RTX PRO 6000

We ran the same image and settings on a desktop RTX 4090, which also drives the screen. At 90% memory utilisation the engine ran out of memory once the desktop took its share, so it runs at 80%.

One copy of the model per GPU, FP8 weights and cache. Code edits use suffix decoding on both; prose uses each GPU's faster setting.

For one person, the gaming card kept up and slightly beat the server GPU on a single code edit (927 tokens per second against 876). With 64 edits at once it delivered 82% of the server card's output. Memory is the real gap. The 4090's cache holds about six typical agent conversations before it starts evicting them; the RTX PRO 6000 holds about 38. One developer or a small team fits on the desktop. A whole team's agents need the server card.

With one GPU there is nothing to route, but Ramjet still earns its place. It gives each person or tool its own API token, tracks usage and cache hits per token, and serves both the OpenAI and Anthropic APIs from one endpoint, with a dashboard on top.

Ramjet's usage view after the desktop runs. Per-token cache hits need vLLM's --enable-prompt-tokens-details flag, which most of this traffic predates; both deployments set it now.

The Anthropic API means Claude Code can drive it. We pointed Claude Code at the 4090 through Ramjet and asked it to fix two planted bugs in a small Python module. Domyn Edge fixed both in 10 turns and 33 seconds, including a median that works for odd and even lengths. pytest was not installed and we had not allowed installs, so it wrote its own test script and said so.

What did not help

• Thinking on for agents: fewer passing tests at three to five times the tokens.
• Guessing 32 tokens instead of 16: no faster on code, and 25-second worst-case waits on prose.
• Tuning suffix decoding's thresholds: prose stayed 31-37% behind.
• A larger prompt-reading step (16K tokens): same throughput, nearly double the wait.
• Triton attention: slower than FlashInfer with both BF16 and FP8 caches.

Running it

Both deployments are single Compose files in the Ramjet repository. Use deploy/domyn_edge for four RTX PRO 6000s and deploy/domyn_edge_4090 for a desktop.

hf download domyn/Domyn-Edge-SFT --local-dir /models/Domyn-Edge-SFT
cd deploy/domyn_edge_4090        # or deploy/domyn_edge
umask 077
printf 'DOMYN_MODEL_DIR=/models/Domyn-Edge-SFT\nDOMYN_API_KEY=%s\nRJ_UI_AUTH_TOKEN=%s\n' \
  "$(openssl rand -hex 24)" "$(openssl rand -hex 24)" > .env
echo 'LB_IMAGE=ghcr.io/helixml/ramjet:rust-f5b7f09@sha256:8a06f192bd13dce3baba429da2fd9a67a6f222760d0794ea996af90e4884e739' >> .env
docker compose build && docker compose up -d

The defaults are the settings measured here, and clients should send reasoning_effort: "none" or leave it out. Every number in this post, including the losing configurations, is in the Ramjet experiment journal under 21 October.

Ramjet is open source under Apache-2.0. Earlier posts cover GLM-5.3 on one 8×H200 server and Ramjet against NVIDIA Dynamo. Talk to Helix about running open European models on your own hardware.