Cutting the memory of a language model: the savings are smaller than the papers suggest

When a language model reads a long prompt, it stores a key and value for every token. This store is the KV cache. At 32,000 tokens it takes 2.4 GB for one query on the model we tested. Many papers propose to throw most of it away and keep only the tokens that matter. We tested that on a free GPU, and asked two questions the papers rarely ask together. Does the model still find the facts it needs? And does the memory really come back?

Short answer: at 25% and 12.5% of the cache, every method we tried lost the needed facts almost every time.

The model was Qwen3-4B-Instruct on a free Kaggle T4. The task was a synthetic one: several project records ("Project X has status S and codeword C") hidden among unrelated documents, then a question about one, two or all three. Answers were graded by exact codeword match. Prompts were 8k, 16k and 32k tokens, 6 cases each. The run was stopped at an agreed time limit, so the 32k tier has 3 of 6 cases. That leaves 255 recorded results out of 306 planned. Each policy and budget below has 15 questions.

We compared six ways of choosing what to keep: no eviction, a sliding window, random tokens, a key-norm score, and our own re-implementations of the SnapKV and H2O scoring ideas. We ran each at token level and at page level (blocks of 16 tokens).

Source: recorded results of the run, 255 records. The 12.5% budget is worse: 0 or 1 correct out of 15 for every method.

With the full cache the model answered 15 of 15. With a quarter of it, the best methods got 2 of 15. The typical wrong answer kept the project name but invented the digits, for example "cobalt-381" instead of "cobalt-2609". Our reading: the scoring keeps the meaning of a token and drops the rare number that carries the answer. That is a hypothesis from these runs, not a tested mechanism.

Inference servers store the cache in pages of 16 tokens. A page can be reused only when all of its 16 tokens are gone. If you delete scattered tokens, you free much less than you think. We simulated this accounting on the same runs. The page-aligned methods free nearly everything, but they also lost the answer. The methods that kept a little accuracy freed far less.

Source: simulated block-of-16 page accounting on the recorded runs (mean of 30 cases per method). Not a measurement on a real paged server. Each scored method logically removed about 1962 MB per question, so 320 MB freed means 16% of that. The window method logically removed 2376 MB per question, and 2373 MB of that became whole pages (the plotted value). Why it removes more than the other methods was not checked, so compare it with care.

One more result. In the Hugging Face static cache we used, decoding did not get faster after eviction (about 1510 to 1640 ms per token). That says the library still attends over the full width. It does not say a paged server would behave the same.

• A 4B open model on a free T4. This says nothing about frontier models.
• SnapKV and H2O here are our re-implementations, with no check against the published numbers.
• Freed pages are simulated. Only the cache allocation was measured on the GPU.
• The 32k tier is half complete. All completed 32k cases also failed.
• One synthetic workload. Real tasks may be more forgiving.

What this supports: a question worth testing properly. A method would need to pick whole pages and keep numbers and names. We have not built one.