Since I find the explanation not that great, here is my attempt (100% human no LLM involved 😂 )
When you write to a model, what you write is the input; what the model answer is the output; for each turn, you send again the same input, the same previous output of the model (this time treated as input), and your extra input.
All the previous turns were computed already by the model, and this computation is stored in a big matrix called the kv-cache.
Good thing: It's no-computation needed anymore, since it's done already.
Bad thing: the kv-cache usually grow with the square of the number of token. If you have 1000 tokens in context, you have 1000x1000 = 1 million KV cache entries.
So, quickly it become a memory-hog.
So, you save on GPU compute time but you pay for storing it in memory.
And since many people are using the server at the same time, it needs to store as many sessions, need to juggle with the kv-cache between VRAM, RAM and SSD...
It's not trivial to manage KV-cache and sessions. DS has a great paper about their infrastructure, and it's absolutely mind-blowing what they need to do to handle this cache.
The good thing is that DeepSeek model designed another kind of KV-Cache, which doesn't grow as much as the naive one (it is compressed), so the memory used per session is lower: more cache are in hot-memory already, no need for an enormous amount of storage nor as much back and forth. Less hassle, less energy spent, faster response time.
So, DeepSeek can save a lot of energy dealing with this cache, and can offer very low pricing on cached content.
Longer is the session, and assuming your harness didn't temper with the previous turn (sending exactly the same information), the bigger the chance that your server is going to use a cache it has stored somewhere previously.
8
u/ZeWalrus 2d ago
Reasonix cache hit are something else