Designing an LLM Response Cache with Redis Cache Aside

Balancing Cost, Delay, and Consistency

This article summarizes how to design an LLM response cache using the Redis Cache Aside pattern.

LLM services may benefit from greater caching than regular APIs. If repeated requests come in with the same question, same document, or same prompt version, there is no need to call the model each time. Reducing model calls reduces latency, reduces costs, and relieves provider rate limit pressure.

However, the LLM response cache can also be dangerous. User permissions may incorrectly reuse answers from other documents, continue to show outdated document-based answers, and even store responses that contain personal information.

Analysis date: 2026-05-12 Practice environment: FastAPI, Redis, PostgreSQL, OpenAI API Main reference materials: Redis Query Caching, Redis TTL/Eviction Docs, OpenAI prompt caching Docs

Key takeaways

• Cache Aside is a pattern that first queries Redis, performs the original task in case of a miss, and then stores it in Redis.
• The LLM response cache must consider ‘question + document scope + prompt version + model + authority scope’ together.
• It is dangerous to cache personal information or authority-dependent responses.
• TTL is a compromise between freshness, cost, and user experience.
• Cache hit rate, saved tokens, and latency reduction should be used as indicators.

1. Why do you need LLM response cache?

LLM calls create three costs:

Cache is very effective for services where the same or similar requests are made repeatedly. For example, official document Q&A, error message explanations, product policy information, and repetitive summary tasks can be cache candidates.

2. Cache Aside pattern flow

The basic flow of Cache Aside is simple.

Copy1# This is an example.21. Receive a request.32. Create a cache key.43. Search the key in Redis.54. If it is a hit, a cached response is returned.65. If it is a miss, call LLM.76. Verify the response.87. Save it with TTL in Redis.98. Return the response.

The advantage of this pattern is that the application directly controls the cache policy. The downside is that poorly designed cache keys and invalidation policies can result in incorrect data.

3. Cache Key Design

The core of the LLM response cache is the key. Do not use only the question text as the key.

Copy1# This is an example.2llm:answer:{hash(question + document_scope + prompt_version + model + user_acl_scope)}

example:

Copy1# This is example code.2import hashlib3import json4 5# This declaration shows an example flow.6def make_cache_key(payload: dict) -> str:7    normalized = json.dumps(payload, sort_keys=True, ensure_ascii=False)8    digest = hashlib.sha256(normalized.encode("utf-8")).hexdigest()9    return f"llm:answer:{digest}"

4. TTL and Invalidation

TTL is the cache survival time. In the LLM response cache, if the TTL is too long, old answers will remain, and if the TTL is too short, the cache effect will disappear.

When a document changes, the associated cache must be invalidated. A simple TTL may cause out-of-date answers to appear immediately after a change.

5. Responses that should not be cached

All LLM responses should not be cached.

Security Caution Cache policies should be designed very conservatively for answers where up-to-dateness and accuracy are important, such as user personal information, full text of internal documents, data with limited rights, and medical, financial, and legal information.

Candidates for banning caches include:

Copy1# This is an example.2[ ] Answers containing sensitive information for each user3[ ] Answer that document access results vary depending on permissions4[ ] Real-time data-based answer5[ ] Frequently changing information such as latest notices, prices, and policies6[ ] Responses indicating that the model is uncertain

6. Cache Stampede response

If caches expire at the same time, many requests may call LLM at once. This can be viewed as a cache stampede.

Here's how to respond:

In initial implementations, just lock and random TTL will work.

7. FastAPI + Redis example

Copy1# This is example code.2from fastapi import FastAPI3import redis.asyncio as redis4import json5 6app = FastAPI()7r = redis.Redis(host="localhost", port=6379, decode_responses=True)8 9@app.post("/answers")10# This declaration shows an example flow.11async def create_answer(req: dict):12    cache_payload = {13        "question": req["question"],14        "document_scope": req.get("document_scope"),15        "prompt_version": "answer.v1",16        "model": "configured-model-name",17    }18    cache_key = make_cache_key(cache_payload)19 20    cached = await r.get(cache_key)21    if cached:22        return {"source": "cache", "data": json.loads(cached)}23 24    answer = await call_llm_and_validate(req)25 26    await r.set(cache_key, json.dumps(answer, ensure_ascii=False), ex=3600)27    return {"source": "llm", "data": answer}

This code is an example to demonstrate the structure. In an actual service, permission scope, personal information masking, timeout, retry, and tracing must be added.

8. Operational indicator design

The important thing about cache is not whether it is attached, but whether it is effective.

If the cache hit rate is low, the key may be too granular or the request pattern may not fit the cache.

9. Practical checklist

Copy1# This is an example.2[ ] Have you classified cacheable and prohibited responses?3[ ] Are prompt_version and model included in the cache key?4[ ] Has the scope of user authority been reflected in the key?5[ ] Are the TTL criteria linked to the document change cycle?6[ ] Is there an invalidation strategy when changing documents?7[ ] Is there a cache stampede response?8[ ] Do you measure cache hit rate and saved tokens?9[ ] Is there a DB/LLM fallback path in case of Redis failure?

10. Q&A

Q1. Are prompt caching and Redis response cache the same?

no. prompt caching is a way for providers to reuse repeated prompt prefix calculations, and Redis response cache is a way for applications to store final responses. The two can be used together.

Q2. Can questions with similar meaning be cached?

It's possible, but the risk is high. Since the semantic cache must manage both the similarity threshold and the risk of wrong answers, it is safer to start with the exact cache initially.

Q3. Should the LLM response cache be stored in the DB as well?

If you need conversation history or audit logs, save them in the DB. Redis is for quick reuse, and DB is for history and analysis purposes.

11. References and uncertainty

References

• Redis Query Caching:https://redis.io/tutorials/howtos/solutions/microservices/caching/
• Redis Docs:https://redis.io/docs/latest/
• OpenAI prompt caching:https://platform.openai.com/docs/guides/prompt-caching

uncertainty

• Cache TTL depends on the service domain and data change cycle.
• An LLM provider's prompt caching policy may vary depending on the model and timing.
• Semantic cache requires separate quality evaluation.

---

Operational example: LLM responses that should not be cached

Redis Cache Aside reduces costs, but applying it to every response is risky. Responses that are highly user-specific or time-dependent, such as personal document summaries, account status, payment information, and latest disclosures, may require narrow cache coverage. The cache key must contain prompt version, model, tenant, permission scope, and input hash, and must not contain the original text of personal information.

A failure case is when questions with similar meaning are grouped with the same cache key. “Is it possible to get a refund?” and “Please tell me the refund process” may seem similar, but the scope of the answer is different. A semantic cache can reduce costs, but it also carries a high risk of reusing incorrect answers. It is safer to start with a deterministic key initially and expand the range after looking at the cache hit ratio and incorrect answer reports.