Self-Hosting Kev on an RTX PRO 6000 With Ramjet

Sep 22, 2026

How we run Kev as an on-premises AI decision model on an RTX PRO 6000, with Ramjet routing TypeSafe System One and OpenAI APIs.

Our eight-GPU inference server now answers two different kinds of request through one Ramjet process. OpenAI clients can call Qwen3.8-Flash-Next or GLM-5.3-Flash. TypeSafe clients can send a document and a set of typed questions to Kev, a small open model built to match the public System One API used by Jev-style decision models.

This is a self-hosted AI deployment in the literal sense: Kev's weights, prefix cache and API stay on a server we operate. There is no external inference hop. Ramjet exposes the decision API through the same authenticated ingress as our language models while preventing either API family from reaching the wrong engine.

Kev does not generate prose. It reads one shared piece of state, answers yes/no, multiple-choice and ordered-score questions, and returns probabilities in one response. That makes it useful for routing tickets, applying policy, scoring urgency and extracting several decisions from the same document.

We fitted the 0.8B model into memory already reserved by a Qwen model copy, then added API profiles to Ramjet so an OpenAI request cannot reach Kev and a System One request cannot reach Qwen or GLM.

Four engines on eight GPUs

Node06 has eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, each with 96 GB of memory. It was already full on paper:

• Qwen3.8-Flash-Next runs as one model copy spread across GPUs 0–3.
• Two GLM-5.3-Flash copies use GPUs 4–5 and 6–7.

The fourth Qwen worker used about 81,216 MiB on GPU 3. That left enough room for kev-0.8b in bf16. Kev added 4,284 MiB of device memory and left about 11.4 GiB free on the card. Its container used 2.868 GiB of host memory.

This is co-location, not a new ninth GPU. Qwen and Kev can both put work on GPU 3. The deployment keeps Kev small, and the performance numbers below include that shared environment.

The other practical constraint was telemetry. Ramjet's static machine view normally assigns each GPU to exactly one upstream. GPU 3 now has two owners, so we left that static ownership map unset instead of publishing a plausible but false diagram.

Ramjet routes the API before it routes the model

Ramjet used to treat every upstream as OpenAI-compatible. The node now has a dense API ownership map next to its model ownership map:

RJ_UPSTREAM_MODELS=qwen3.8-flash-next,glm-5.3-flash,glm-5.3-flash,kev-latest
RJ_UPSTREAM_APIS=openai,openai,openai,systemone

For every request, Ramjet first identifies the route family. /v1/systemone selects the System One profile. OpenAI paths select the OpenAI profile. It then intersects that profile with the requested model before choosing an upstream.

The intersection matters during failures as well as normal routing. Retries and fail-open behavior stay inside the same API profile. A request for kev-latest on an OpenAI endpoint returns 404 without contacting Kev. A System One request naming Qwen does the same without contacting Qwen.

Model discovery also keeps its original contract. GET /v1/models returns a normal OpenAI model list containing Qwen and GLM. Ramjet probes Kev's private GET /v1/models response for readiness, but does not splice its different schema into the public OpenAI response.

The result is one public ingress and one set of health and routing metrics. It is not a protocol translation layer. Ramjet preserves the native request and response bodies for each API.

Kev stays on a private network

The upstream Kev server binds to loopback by default and has no authentication. Our derivative image changes only the bind host so the process can listen inside a fixed internal Docker network. The container publishes no host port.

Kev runs as user 1000 with a read-only root filesystem. Its model cache is mounted separately, and both Hugging Face Hub and Transformers run in offline mode after the pinned checkpoint has been downloaded. The deployed source commit, adapter revision and image digest are fixed in the deployment record.

External clients still enter through authenticated Caddy and the same Ramjet port used by OpenAI traffic. Ramjet is the only container connected to all four engine networks.

How to self-host Kev behind Ramjet

The production shape has three boundaries: Kev owns the System One contract, Ramjet owns routing and health, and Caddy owns public TLS and authentication. Kev is not exposed directly.

The core Ramjet configuration is a pair of dense, position-matched maps:

RJ_UPSTREAM=http://qwen:8000,http://glm-b:8000,http://glm-c:8000,http://kev:8009
RJ_UPSTREAM_MODELS=qwen3.8-flash-next,glm-5.3-flash,glm-5.3-flash,kev-latest
RJ_UPSTREAM_APIS=openai,openai,openai,systemone

For an on-premises deployment, the practical checklist is short:

1. Build Kev from a pinned source revision and pre-populate a persistent Hugging Face cache with the pinned model revision.
2. Put Kev and Ramjet on an internal container network. Publish Ramjet, not the Kev port.
3. Register the Kev upstream with model=kev-latest and api=systemone at the same ordinal.
4. Put authentication and TLS in front of Ramjet, then test successful calls and deliberate cross-profile 404s.
5. Benchmark with your real state length and question mix. Prefix-cache results depend on repeated state, not just model size.

Our checked-in node06 deployment includes the Compose stack, validators, immutable image inputs, warm-up request and rollback script. It is a concrete reference for self-hosting Kev beside existing OpenAI-compatible models rather than a generic one-container quickstart.

Performance on the RTX PRO 6000

We sent 85 synthetic requests through Ramjet. Each request contained one choice, one noul and one score question. All 85 returned HTTP 200, all reached Kev at Ramjet upstream ordinal 3, and every response contained the three expected answer types.

Kev serializes model execution with one lock. Four simultaneous requests improved aggregate throughput by keeping CPU and HTTP work moving around the GPU pass, but each request spent longer waiting its turn. For an interactive caller, concurrency 1 gives the lower latency. A queue worker can use modest concurrency to get more decisions per second.

The long-state result shows why the prefix cache is useful for document workflows. The first 5,524-token request took 291.0 ms end to end. Kev retained the state activations, then answered the next 12 sequential requests in 87.7 ms at the median. That is a 69.9% reduction in end-to-end latency. The server recorded one cache miss followed by 36 hits across the sequential and concurrent cached cells.

These are live co-location measurements. Two unrelated Qwen requests used the shared four-GPU model copy during the eight-second run, and the test did not try to find the card's saturation limit. The intake-air guard passed at 41°C, GPU temperature peaked at 62°C, and no container restarted. The figures describe this node, checkpoint and request shape.

Self-hosted Kev API examples with curl

The JSON contract accepts a string, object or array as state. Question IDs belong to the caller. One request can mix all three question types:

curl https://inference.example.com/v1/systemone \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kev-latest",
    "state": {
      "ticket": "Shoes arrived late and in the wrong size. I also see two card charges.",
      "account_tier": "business"
    },
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "returns": "Exchanges, refunds, wrong or damaged items",
          "shipping": "Delivery status, delays or lost packages",
          "billing": "Charges, invoices or payment problems"
        }
      },
      "escalate": {
        "type": "noul",
        "instructions": "Does this need prompt human attention?"
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this ticket?",
        "criteria": ["can wait", "this week", "today"]
      }
    }
  }'

A noul answer contains the probability of yes. A choice answer contains the selected option, confidence and probability for every option. A score answer contains the expected numeric level, its legend and the level probabilities. Kev computes decisions; usage.output_tokens counts the serialized answer and is not a text-generation token count.

For document classification, send the extracted text as state and ask one constrained routing question:

curl -sS https://inference.example.com/v1/systemone \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kev-latest",
    "state": "Invoice 8841 requests payment for replacement laptop batteries.",
    "questions": {
      "document_type": {
        "type": "choice",
        "instructions": "Classify this document for the intake queue.",
        "criteria": {
          "invoice": "A bill or request for payment",
          "purchase_order": "Authorization to buy goods or services",
          "contract": "Terms governing an agreement",
          "other": "None of the listed document classes"
        }
      }
    }
  }' | jq '.answers.document_type'

For an agent action gate, ask for a probability instead of free-form approval text. The application still owns the threshold and the fallback:

curl -sS https://inference.example.com/v1/systemone \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kev-latest",
    "state": {
      "requested_action": "refund_order",
      "refund_usd": 740,
      "policy": "Agents may auto-refund up to $250. Larger refunds require human approval."
    },
    "questions": {
      "human_approval_required": {
        "type": "noul",
        "instructions": "Does policy require a human to approve this action?",
        "criteria": {
          "true": "The action requires human approval",
          "false": "The action can proceed automatically"
        }
      }
    }
  }' | jq '.answers.human_approval_required'

These examples use one question for clarity. In production, grouping related questions around the same state avoids resending a document and lets Kev reuse one model pass.

Application patterns for an on-premises decision model

Kev fits work where the input is rich but the output should be bounded and machine-readable:

• Customer-support triage: choose a queue, score urgency and estimate whether a human escalation is needed in one request.
• Private document routing: classify invoices, contracts, policies or correspondence without sending the source text to a hosted inference provider.
• Agent action gates: return a probability for whether a proposed refund, account change or tool call meets a written policy. The calling application applies its own threshold and routes uncertain or high-impact cases to a person.
• Repeated analysis of a long state: ask several typed questions about the same handbook, case file or evidence bundle, then benefit from Kev's prefix cache when that state is queried again.

The bounded schema is useful, but it does not make a probabilistic model authoritative. High-impact legal, financial, employment or clinical decisions still need domain-specific validation and human review.

Calling it with the TypeSafe SDK

Kev implements the public System One contract, so the TypeSafe Python client can point at the same Ramjet base URL:

import os
 
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
 
client = TypeSafeClient(
    api_key=os.environ["API_KEY"],
    base_url="https://inference.example.com",
    model="kev-latest",
)
 
response = client.system_one(
    state="I was charged twice. Please fix this today.",
    questions={
        "billing": Noul(instructions="Is this ticket about billing?"),
        "tone": Choice(
            instructions="What is the customer's tone?",
            criteria={"calm": None, "frustrated": None, "angry": None},
        ),
        "urgency": Score(
            instructions="How urgent is this ticket?",
            criteria=["can wait", "this week", "today"],
        ),
    },
)
 
print(response.nouls["billing"].noul)
print(response.choices["tone"].choice)
print(response.scores["urgency"].score)

The same base URL still serves OpenAI clients

Nothing changes for an existing OpenAI caller. The model name keeps the request inside the OpenAI profile:

import os
 
from openai import OpenAI
 
client = OpenAI(
    api_key=os.environ["API_KEY"],
    base_url="https://inference.example.com/v1",
)
 
response = client.chat.completions.create(
    model="qwen3.8-flash-next",
    messages=[{"role": "user", "content": "Summarize this incident."}],
    max_tokens=256,
)
print(response.choices[0].message.content)

The deployment now gives generative and decision workloads one operational surface while keeping their contracts separate. Qwen and GLM retain their existing engine projects and caches. Kev can be stopped or replaced without restarting either large model.

The Ramjet source and deployment, Kev source, and Kev-0.8B weights are public.

Measured on jaredpalmer/kev-0.8b revision 54f4f8777356cd5bbbb6c6919c657f26e6f2f6d8, backed by Qwen3.5-0.8B-Base in bf16, on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU shared with a Qwen TP4 rank. Requests passed through the same Ramjet process used by production OpenAI traffic.