What does a cache hit mean in SillyTavern?

By Arscelare
On this page

A prompt-cache hit means the model service reused processing for part of your input. It may reduce the charge for that part and the time spent processing the prompt. The service still generates a new reply.

SillyTavern assembles the request; the selected provider and model determine how caching works. Seeing the same character card on two turns does not, by itself, guarantee a reported hit.

What the service can reuse

In prefix-based prompt caching, the reusable part is an unchanged beginning of the prompt. That beginning might contain stable instructions, a character description and earlier messages.

Consider this simplified example. The labels describe prompt sections, not text to paste into a preset:

Earlier request:
[Instructions] [Character] [Earlier messages] [Your message A]

Next request:
[Instructions] [Character] [Earlier messages] [Your message A]
[Model reply A] [Your message B]

The next request extends the same beginning. If that prefix is eligible and its cache is available, the service can reuse processing for it. The new material still needs processing, followed by generation of the next reply.

An early edit can shorten the matching prefix. Changing the character description, for example, can prevent reuse of the conversation after it even when those later messages are unchanged. A shorter matching section before the edit may remain reusable.

Both OpenAI's caching documentation and Claude's caching documentation describe prefix reuse. Their eligibility rules, controls, lifetimes and prices differ. Those details are not interchangeable between models or providers.

A cache hit does not replay an old answer

Prompt caching reuses work on the supplied input. It does not mean the service retrieves an old reply instead of writing one for your new message. Repeated wording in replies is not evidence of a cache hit; check the reported usage.

Cached input still occupies context. If 8,000 of a request's 10,000 input tokens are cached, the prompt is still 10,000 tokens long. Caching does not make extra room for older messages.

It also does not restore scene details that SillyTavern left out of the request. A character cannot rely on omitted history simply because some other part of the prompt was cached. Keep chat storage, active context and processing reuse separate when diagnosing memory problems.

Prompt caching itself is not a roleplay-quality setting. Changing your preset or removing useful lore to increase hits can change the writing because you changed the input. Judge that tradeoff by the resulting scene, not the cache percentage alone.

Why a later turn might miss the cache

Check these possibilities before assuming the feature is broken:

  • The provider or selected model has different caching requirements from the one you used before. Some require explicit cache controls; others apply caching automatically.
  • The reusable prefix does not meet the model's eligibility requirements, such as a minimum length.
  • A note, timestamp, macro or activated lore entry changed an early part of the prompt.
  • A summary, retrieval result or context-window change altered which history was sent.
  • Too much time passed, or the service no longer has the matching cache available.
  • Your client does not display the provider's cache-usage fields, so the visible information is incomplete.

SillyTavern's Chat Vectorization documentation warns that dynamic prompt content can disrupt prefix reuse. That is a reason to inspect the assembled request, not proof that every lorebook or retrieval setup prevents caching.

Keep facts the scene needs. Deleting useful context merely to raise the hit rate can leave you with a cheaper request and a worse conversation.

Read a partial cache hit without counting input twice

Billing fields differ between providers. Some report total input with cached input as a subset; others separate uncached input, cache reads and cache writes. Read the definition for your service before adding the numbers.

Arscelare reports total input as prompt_tokens and its cached subset as prompt_tokens_details.cached_tokens. Subtract the subset once to find uncached input. The cached-input rate applies only when the model server reports a hit.

Here is hypothetical arithmetic using Arscelare's listed rates. Assume 10,000 total input tokens, of which 8,000 are reported cached, and 1,000 total output tokens including reasoning:

Billed categoryTokensUSDC per million tokensCharge
Uncached input2,0001.500.003 USDC
Cached input8,0000.3750.003 USDC
Total output1,0003.000.003 USDC
Total charge0.009 USDC

With zero reported cached input, the same assumed input and output counts cost 0.018 USDC at those rates. This compares two invented requests; it is not a measured hit rate, savings forecast or promise about your next turn.

Do not apply this formula unchanged to a provider with separate cache-write charges. Claude's documentation, for example, distinguishes creating a cache entry from reading one. For a worked comparison of short and long histories, see why SillyTavern chats get more expensive as they grow.

Check your own setup one change at a time

First, identify the exact model, provider and SillyTavern connection source. Consult that provider's current caching instructions. A setting for one connection does not establish how another connection behaves.

Inspect the prompt with SillyTavern's Prompt Itemization or another documented inspection method. Compare the beginning of successive requests, especially the instructions and material inserted before history. Keep any logs private; they can contain chat text or keys.

Use a few ordinary turns to compare total input, reported cached input, output and actual charge. Extra test generations may be billed. Faster generation alone is not enough evidence of a hit, and one zero reading does not establish that caching never works.

If useful scene notes change often, accept that some reuse may be lost. Avoid adding filler just to reach a cache threshold. Measure the total bill and the writing you keep.

If you want to use Arscelare, the SillyTavern setup guide covers the connection. Use the reported usage from your own requests when deciding what a conversation costs.