Ramjet Now Mints API Tokens and Counts Every Request

Oct 20, 2026

Ramjet, our open-source load balancer for vLLM and SGLang, can now issue its own API tokens and attribute every request to a token and a model. The first usage store needed 5.1 GiB of RAM for 1,000 active tokens. The SQLite version needs 7-25 MiB, and Parquet files measured about six times smaller again.

Our first design for per-customer usage tracking needed 5.1 GiB of RAM to follow 1,000 API keys. Traffic had almost nothing to do with it. A key that sends a billion tokens a day costs a few counter increments. A thousand keys that each send one request an hour cost a thousand new rows every hour, each carrying its own latency histograms. We measured that before shipping and threw the design away. The version now running in production would track the same 1,000 keys in 7 to 25 MiB.

We needed that tracking because every client of our production inference server shared one key. Ramjet is the open-source load balancer between those clients and the GPUs: every request passes through it, and it decides which copy of the model answers. Until this release, a reverse proxy in front of it checked the one shared key. We could see how hard each GPU was working and which prompts it had cached, but not whose work it was. A helpful agent and a runaway batch job looked the same.

Ramjet can now mint a key per client and attribute every request to a key and a model: requests, prompt and completion tokens, cache hit ratio and latency, by minute, hour or day. Making that history cheap to store took three designs, and one of our benchmarks overstated the best of them by a factor of 22.

Minting a token

A token is created with one call to Ramjet's management API, authenticated with the dashboard's admin token:

curl -s -X POST -H "Authorization: Bearer $RJ_UI_AUTH_TOKEN" -H 'content-type: application/json' \
  -d '{"name":"helix-prod","description":"Helix agent fleet"}' http://127.0.0.1:8007/api/tokens

The response contains the secret, once. A client then uses it like any other key against the OpenAI-compatible endpoint:

curl -s http://ramjet.example.com/v1/chat/completions \
  -H "Authorization: Bearer rjk_..." -H 'content-type: application/json' \
  -d '{"model":"qwen3.8-flash-next","messages":[{"role":"user","content":"Hello"}]}'

A token looks like rjk_<12-hex id>_<64-hex secret>. The id is public: it is the label in the dashboard and in metrics. Ramjet stores only the SHA-256 of the secret, so a copy of its state file cannot be replayed as a credential. Each token has a name, a description and an optional expiry, and can be edited, revoked or deleted. Ramjet accepts it as Authorization: Bearer or, for Anthropic-style clients, as x-api-key. One instance holds up to 10,000 tokens.

The engines never see a minted token. After checking it, Ramjet removes the client's credential and forwards the request with the original master key, the one the engines were started with. That master key also keeps working for clients, so nothing that already called the server had to change.

The Tokens tab, rendered from the dashboard's built-in mock data. Search, a status filter and paging keep it usable at thousands of tokens.

On our production server, node06, authentication has moved from the Caddy reverse proxy into Ramjet, keeping the same key. Before switching, we ran the new build for ten minutes with authentication off so it only recorded who sent each request: every request carried the key, so no client had to change. Now minted tokens reach Ramjet, and a rejected request is answered by Ramjet with a 401 and counted.

What gets counted

Accounting is on by default, whether or not tokens are in use. Every request is attributed to a principal: a minted token, the master key, anonymous when authentication is off and no known key was sent, or rejected when authentication failed. Each principal is broken down by model and time:

• Time buckets: minutes for the last 24 hours and hours for the last 90 days. Day views are built from the hours, aligned to midnight in the viewer's time zone.
• Tokens: prompt, cached prompt and completion, plus the cache hit ratio and how many requests were fully cached, partly cached or cold.
• Latency: time to first token at p50, p90 and p99, total duration, generation speed and time per output token.
• Outcomes: client and server errors, client disconnects, finish reasons, and which engine served the request.

Ramjet also keeps the last 100,000 individual requests. Each record holds counts, timings, status and routing outcome. No prompt or response is ever recorded.

For alerting, two Prometheus counters carry the same attribution: ramjet_principal_requests_total and ramjet_principal_tokens_total, labelled by principal and model. The principal label is the token id, never the secret.

The dashboard

Clicking a token opens its own page: usage over time, a per-model breakdown, finish reasons, and that token's slice of the request log.

The Usage tab compares principals. It groups by token, by model or not at all, and shows the busiest tokens individually with the rest folded into one "other" group. Its counters and latency histograms are exact, not sampled, because they are computed by subtracting the named groups from the fleet total.

All screenshots use the dashboard's synthetic mock data, not production traffic.

First version: everything in memory

The first implementation kept every bucket in memory and rewrote a JSON snapshot to disk every minute. To size it, we filled it as if every token had been active every hour for 90 days and every minute for the last day:

At 1,000 tokens, ranking the tokens over 30 days took 2.0 seconds.

The cost grew with the number of distinct tokens that were active in each bucket. Traffic volume barely mattered: a token sending a billion LLM tokens a day only increments a few counters, while a thousand tokens each sending one request an hour create a thousand rows every hour. The limit is 10,000 tokens.

Second version: SQLite and a delta map

The rewrite keeps the request path away from the database. A finished request only updates an in-memory map of changes since the last flush. Every 10 seconds, a background task merges those changes into SQLite rows, one per tier, principal, bucket and model, in write-ahead-log mode. A crash loses at most one interval, and a database failure is logged and retried without failing a request.

Two choices keep reads cheap:

• Every request is also recorded under a reserved principal, *, so the whole fleet and each model are pre-aggregated. Fleet and per-model charts read one row per bucket, however many tokens exist.
• A single token's page reads only that token's rows, through the primary key.

Memory now depends on how many keys change within 10 seconds, not on history or on how many tokens exist.

Measuring a columnar store

SQLite stores each bucket as a row with its keys, its counters and its latency histograms, plus an index. We measured the alternatives on realistic data: randomized traffic over 31 days for 100 and 1,000 tokens.

SQLite used about 440 bytes per row including keys and index. Parquet used about 74, roughly six times less. Compressing each row with zstd inside SQLite made the database about half the size, a smaller gain. On this data set, ranking 1,000 tokens over 30 days took 95 ms in SQLite and 3.7 ms with DuckDB reading the Parquet files.

The request log showed the same pattern: about 413 bytes per request in SQLite and 56 in Parquet.

Our first comparison put Parquet 133 times ahead. That test generated identical synthetic requests, and a columnar format compresses a column of repeated values to almost nothing. With randomized token counts, timings and models, the advantage fell to about six times. The six is the number we are planning around.

We also looked at Apache Iceberg and decided against it. Iceberg is a table format for many engines sharing data on object storage, with a catalog, manifest files and compaction jobs. Ramjet is a single writer on a local disk and needs none of that. A team that wants this data in a lake can copy the Parquet files there.

Next: a tiered store

This is planned and not built. SQLite would keep the recent and still-open buckets, where rows change every 10 seconds. Each closed day would be compacted into an immutable zstd Parquet file: written to a temporary file, fsynced, renamed into place, and only then deleted from SQLite. Daily rollups would be kept long-term, and anyone could query the files directly with DuckDB.

It has a cost. We measured that Arrow and Parquet would add about 9 MB to Ramjet's 15.7 MB binary and about 48 seconds to the one-time dependency build. The request path would not change: requests still only touch the in-memory delta map.

Ramjet is open source under Apache-2.0. For how it chooses an engine for each request, see Ramjet vs NVIDIA Dynamo. Talk to Helix about running open models on your own hardware with per-team keys and usage reporting.