Shared Reserved Inference
A third way to serve open models — one GPU split into equal, guaranteed lanes for a bounded cohort, billed by the hour instead of the token. Here's how it performed under 32 real users running DeepSeek‑V4‑Flash‑0731 for an hour.
Live run2026-08-18, 13:41–14:41 UTCDeepSeek-V4-Flash-0731 (full weights, 1M context)measured, not modeled
A third way to serve a model
Hosted inference forces a bad choice. Per-token APIs once looked cheap and elastic — but rates now move without warning, providers throttle or retire models, there's no guaranteed throughput, and at real volume the metered bill quietly overtakes the cost of the GPU actually doing the work. The obvious fix — rent your own GPU— flips the problem: a dedicated node bills every hour whether you use it or not, which no individual or small team can keep busy enough to justify. So you're stuck between an API you don't control and hardware you can't afford to leave idle.
Shared Reserved Inference splits one GPU into equal, guaranteed lanes across a small, fixed cohort. A handful of users share the cost of one warm, well-utilized node — each paying a flat hourly fraction — while keeping the guarantees a shared API can't give.
| Per-token API | Shared Reserved | Dedicated GPU | |
|---|---|---|---|
| Who's on the GPU | Anonymous shared pool | A fixed cohort, equal lanes | One tenant, whole GPU |
| Guarantee | None — best-effort | Guaranteed lane + burst | Full GPU |
| Pricing | Per token; rates shift, models retire | Flat hourly, no surcharge | Per GPU-hour |
| Cost to user | Volatile; overtakes GPU cost at volume | Low, fixed — a fraction of a node | High — you pay for idle time |
| Warm cache | Fragmented across replicas | Sticky — stays hot | Sticky |
The result is dedicated-grade guarantees at shared-grade price: your lane is always there, the cache stays warm, and the bill doesn't move when the world gets busy.
The benchmark setup
We put 32 real users on a single reserved node for an hour, each with their own API key, running coding-agent workloads through a standard CLI harness.
What one hour looked like
The cache is the engine
Agentic coding re-sends the same repo, system prompt, and history on every turn. A sticky lane keeps that context warm, so 97% of all input tokens were served from cache — skipping the expensive prefill step. That hit rate held steady across the whole hour, under full load, and warm requests started returning tokens in about a second.
Value delivered per user
Every token each user consumed was priced at DeepSeek's own published API rates, so the numbers are directly comparable to what the same usage would have cost on the first-party API. Against a flat $0.20/hour lane:
Average per-user value for a $0.20 lane-hour
DeepSeek-equivalent token value consumed, one hour, per user — dashed line marks the $0.20 lane cost
DeepSeek doubles all token rates for ~7 hours a day; a flat lane never does — so the same usage is worth 1.7× the lane price off-peak and 3.4× at peak. Usage was evenly distributed across the cohort, from lighter to heavier users:
Per-user value distribution (32 users, off-peak basis)
Each bar is one user's DeepSeek-equivalent value consumed in the hour
Average $0.34, median $0.35, range $0.17–$0.52 off-peak (double those at peak). The heaviest user pulled 2.6× off-peak / 5.2× peak value from a single $0.20 lane.
The saturation ceiling
Across the live hour, the 32-lane cohort used only about a third of the node's capacity — real demand, with clear headroom to grow. To establish the full ceiling we saturation-tested the node separately at 64 concurrent streams with speculative decoding on. The result: a warm cache doesn't just cut cost, it raises throughput.
| 64-concurrent profile | Output tok/s | Per user | Total tok/s |
|---|---|---|---|
| ~98% cache hit (production-like) | 6,206 | ~97 | 107,000 |
| Cache miss (fresh prompts) | 3,291 | ~51 | 29,600 |
| Very-long-context, uncached (64:1) | 551 | ~9 | 35,800 |
Warm cache nearly doubled sustained output (6,206 vs 3,291 tok/s) by freeing compute from prefill and redirecting it to generation. At the production-like ceiling, 64 lanes each sustain ~97 tok/s — and the live run's throughput sat at only ~30% of this, confirming the node carried its cohort with headroom to spare.
- Measured, not modeled. All figures are from live telemetry over the exact window 2026-08-18 13:41:49–14:41:49 UTC: per-key token and spend logs, and the serving engine's own cache/throughput counters. Token value is computed at DeepSeek's published off-peak rates ($0.22 input / $0.007 cached / $0.66 output per 1M); peak figures apply DeepSeek's 2× peak schedule.
- One workload class. The cohort ran coding-agent traffic (large, stable contexts; short outputs), which is cache-favorable by nature. Cache hit rate and per-user value will differ for one-shot or low-reuse workloads.
- The live run used ~30% of node capacity, so its per-user values reflect real demand, not the ceiling. Ceiling figures come from a separate 64-concurrent saturation benchmark (cache-miss and ~98%-hit datasets, speculative decoding on, driven on the node).
- $0.20/lane-hour is an illustrative price, not a cost disclosure. Value multiples compare user-consumed DeepSeek-equivalent value to that price.