An open-weight model can be served by many providers. The weights may be the same, but the prompt cache in front of them is not.

I sent equivalent, roughly 4,000-token prefixes through 23 providers on OpenRouter, left them untouched, and then requested them again. Depending on the provider, a cached prefix survived for less than 30 seconds or more than 24 hours. That spread appeared even between providers serving the same model.

I measured retention with a prime → verify → idle → probe cycle. Each trial sent a unique prefix, checked ten seconds later that the cache write was visible, left the prefix unread for a fixed interval, and then sent it once more. The final response's cached_tokens field revealed whether the prefix had survived. The results below cover retention trials across DeepSeek-V4-Flash and GLM-5.2.12

Cache writes become visible quickly#

Retention only matters after a cache write succeeds. To separate slow writes from early evictions, I ran a second sweep that re-probed fresh prefixes at fixed delays from 100 milliseconds to 60 seconds. Each cell below shows the fraction of writes visible after that delay. Rows that brighten from left to right indicate propagation; persistently dark rows indicate writes that were never observed.

Cache-write visibility heatmap for open-weight providers
Cache-write visibility after the prime. Most providers exposed the cached prefix within two seconds; several had persistently lower hit rates, consistent with routing or load-balancing noise.

22 of 23 providers crossed 50% visibility within two seconds. deepseek-parasail was the only provider that did not expose a write within the 60-second window. For most providers, an application can reuse a prefix immediately; inserting an artificial delay is unlikely to improve its hit rate.

The provider determines how long—and how reliably—the cache survives#

The simplest way to compare providers is effective TTL: the longest tested idle interval before verified retention first falls below 50%. I swept intervals from 30 seconds to 24 hours. The result is an observed threshold under live load, not a provider guarantee.

The range is striking. DeepSeek's native provider held DeepSeek-V4-Flash for six hours; every other provider serving that model had an effective TTL of two minutes or less. For GLM-5.2, glm-streamlake stayed warm for the full 24-hour observation window, while several providers fell below 50% retention before 30 seconds.

But effective TTL hides an important distinction: two providers can cross the 50% line at the same interval and still fail very differently. The matrix below shows the full retention curve. Each column is an idle interval, and each cell is the fraction of verified writes that survived it.

Retention probability by idle interval for every open-weight provider
Verified retention probability per idle interval, rows sorted by effective TTL; gray cells had no verified sample.

Some rows form a cliff: retention stays near 100% and then collapses within one or two intervals, consistent with a fixed expiry timer. Others are ragged: hit probability degrades unevenly with no clean boundary, consistent with capacity pressure, routing noise, or other load-dependent eviction.

This is why there is no useful model-level cache TTL. The unit of comparison is a specific model–provider pair, and the right provider depends on the workload's idle gaps. Gaps that sit comfortably before a stable cliff can reasonably be budgeted as warm. Gaps that straddle the cliff—or land on a ragged row—should be budgeted for a possible full prefill.

Retention also changes with load#

The same provider did not behave identically throughout the day. Breaking effective TTL down by the UTC hour in which the prefix was primed reveals which providers stayed in one retention tier and which shifted as operating conditions changed.

Effective TTL by hour of day for open-weight providers
Effective TTL by prime-time window (UTC), split by model; each column pairs two hours and reports the weaker of them, and color encodes bucketed retention tiers so a tier change across the day stands out.

These windows contain fewer samples than the aggregate sweep, so they are better read as a stability check than as a scheduling guide. If a provider changes tier during the hours that carry most of your traffic, use those weaker hours for capacity planning rather than relying on its aggregate TTL.

How much of the prompt actually counts#

TTL measures how long a cache entry survives. Cache depth measures how much of the prompt receives cached-token credit when it does survive. I calculate depth as cached_tokens / prompt_tokens on retained hits. Values just below one can result from block-aligned caching; lower values mean part of the prompt is still processed as uncached input.

Credited cached fraction on retained hits, by model and provider
Cache depth on retained hits: long-lived is not the same as deep — glm-streamlake retains longest yet credits a shallower fraction than many shorter-lived providers.

Most providers credited nearly the entire prompt on a hit, but depth and retention did not move together. glm-streamlake had the longest retention in the study and a shallower credited fraction than several providers that expired within minutes. A realistic cost estimate therefore needs both the probability of a hit and the fraction of input tokens credited on that hit.

A billing hit is not always a latency hit#

A cached prefix can reduce prefill work, so the warm probe may return its first token faster than the cold prime. I measured time to first token on the same streamed requests used by the retention probe.

First-token latency, cold prime versus warm probe, per provider
Cold-prime versus warm-probe TTFT per provider; the latency payoff exists only where the warm box sits clearly below the cold one.

Some providers credited cached tokens without a clear reduction in first-token latency. In fact, in many cases, cache hits can be slower due to queueing and prefix cache transfer. On those providers the cache may lower the bill without making an interactive request feel faster. Billing and latency are separate outcomes and should be measured separately.

Choose the serving provider as carefully as the model#

For cache-heavy workloads, selecting open weights is only half the deployment decision. An agent that repeatedly sends a long system prompt, a chat service with shared conversation history, or a RAG pipeline with a stable prefix should evaluate each model–provider pair on three independent axes:

  1. Retention: Does the provider stay warm across the workload's actual idle gaps?
  2. Depth: How much of the prompt receives cached-token credit on a hit?
  3. Latency: Does a retained hit materially reduce time to first token?

Provider price, output quality, rate limits, and availability still matter. But a model's advertised cached-input price is not enough to predict the bill: the provider must retain the prefix long enough to earn that price in the first place.

The proprietary-model companion post applies the same measurement to OpenAI, Gemini, Anthropic, and xAI, where the model and serving stack cannot be separated.

These measurements were collected through OpenRouter in early July 2026, not with direct provider keys. Prefix caching is best-effort and load-dependent, so exact TTLs can drift between runs and across hours. Read the results as retention tiers, not contractual limits.

Footnotes

  1. The measured idle interval is verify → probe with no reads in between, so a miss means the prefix genuinely sat idle that long, not that a refresh confounded it. Retention is verified-only: it is computed only over trials whose prime provably cached by the +10s verify probe, so a provider that never landed warm is excluded rather than scored as an eviction. Effective TTL is a first-crossing rule — the last idle interval before verified retention drops below 50%, scanning shortest-first, so a lone hit at a longer interval does not revive it. ≥ means window-limited: the cache survived the largest interval I observed for that provider, and the true TTL may be longer. Retention tracks backend load, not just elapsed time, so exact numbers shift between runs; longer intervals are sometimes skipped after confirmed short-interval misses, which tilts long-interval estimates toward healthier cycles. And OpenRouter's prefix stickiness is not guaranteed across hour boundaries, one source of verify/probe noise. ↩

  2. Each trial starts with a fresh 32-character random nonce written at the very front of the prompt as EXPERIMENT-NONCE: <nonce>. Front placement is deliberate: a nonce at the end would let different trials share a long common prefix and prime each other's cache. After the nonce, the prefix is filled to the provider's character budget with deterministic filler sentences drawn from a fixed pool by a per-trial seed, then truncated exactly at the budget. This keeps every prime/probe pair unique while holding token shape comparable across trials. ↩