Prompt Caching and Token Budgets

Operational strategies to reduce costs and delays

In this article, we summarize how to manage the cost and delay of LLM services as operational indicators using prompt caching and token budget.

LLM costs are not fixed like server CPU costs. Each request depends on the input token, output token, model, and whether or not it is cached. The same goes for response delays. Even within the same API, the length of the prompt and the number of documents retrieved can vary greatly.

Therefore, in LLM services, “how to manage prompts as an operable cost unit” is more important than “how to write prompts well.”

Analysis date: 2026-05-12 Practice environment: OpenAI API, FastAPI, Redis, Prometheus Main reference materials: OpenAI prompt caching Docs, OpenAI Rate Limits Docs, OpenAI Cookbook

Key takeaways

• token budget is the input/output token limit allowed for one request.
• prompt caching helps reduce the cost and delay of repeated prompt prefixes.
• Placing static content at the front of the prompt and dynamic content at the back is advantageous for cache efficiency.
• Cost optimization should be based on usage metrics, not intuition.
• Cost and quality for each prompt version must be looked at together.

1. Why LLM costs are difficult to predict

For regular APIs, the cost can be somewhat estimated by looking at the number of requests and server resources. But with LLM, the length of the request creates a cost.

Therefore, token usage should be viewed as an API metric.

2. What is token budget?

token budget is the token budget to allow for one request.

Copy1# This is an example.2Total budget = system prompt + tool schema + retrieved context + user question + output reserve

example:

If you exceed your budget, you may need to reduce the number of search documents, summarize old conversation history, or limit output max tokens.

3. prompt caching basic concept

prompt caching is a method in which the provider reuses repeated prompt prefixes to reduce delays and costs.

What is important in practice is that “repeated parts should be at the front.”

Copy1# This is an example.2Good structure:3[fixed system prompt]4[Fixed tool schema]5[Fixed response rules]6[Dynamic Search Document]7[Dynamic User Questions]8 9Bad structure:10[User Question]11[Search Document]12[fixed system prompt]13[tool schema]

Cache prefix reuse may be less effective if the fix content is located later.

4. Cache-friendly Prompt Layout

The recommended layout is as follows.

Copy1# This is an example.2System:3- Role4- Reply Policy5- citation rules6- Prohibitions7- Schema description8 9Developer:10- Tool use rules11- Output validation criteria12 13Context:14- retrieved chunks15 16User:17- question

Policies that do not change are placed first, and documents and questions that change with each request are placed last.

5. Divide token budget in RAG

For RAG, token budget management is especially important. Entering more search results does not always result in better answers. Rather, if the number of unrelated chunks increases, the model may become confused.

6. Cost/delay indicator design

Be sure to leave metrics:

Copy1# This is an example.2llm_input_tokens_total3llm_output_tokens_total4llm_cached_input_tokens_total5llm_cost_estimated_total6llm_latency_ms_bucket7llm_prompt_version_usage_total8llm_rate_limit_error_total9llm_retry_total

Analysis criteria:

7. Rate Limit and Backoff

Rate limits are not a problem that can be solved by simply using a higher rate plan. Combining burst traffic, retry storms, and bulk eval executions can lead to throttling.

Response principles:

Copy1# This is an example.2[ ] Request unit rate limit3[ ] User/Organizational Unit Quota4[ ] eval batch separate restrictions5[ ] exponential backoff6[ ] retry budget7[ ] provider fallback

8. Practical example

Budget check before calling prompt:

Copy1# This is example code.2def enforce_token_budget(parts: dict, max_input_tokens: int):3    estimated = sum(estimate_tokens(value) for value in parts.values())4    if estimated <= max_input_tokens:5        return parts6 7    parts["retrieved_context"] = trim_context(parts["retrieved_context"])8    estimated = sum(estimate_tokens(value) for value in parts.values())9 10    if estimated > max_input_tokens:11        raise ValueError("TOKEN_BUDGET_EXCEEDED")12 13    return parts

In actual services, tokenizer-based estimation and provider usage results must be recorded together.

9. Practical checklist

Copy1# This is an example.2[ ] Have you set an input/output token budget per request?3[ ] Do you record token usage by prompt version?4[ ] Did you place the static prompt prefix in front?5[ ] Are the RAG context top-k and maximum tokens limited?6[ ] Have you secured the output reserve?7[ ] Do you measure the cached input token ratio?8[ ] Are rate limit errors and retry costs recorded?9[ ] Have you separated quotas for eval execution and user traffic?

Failure case: Cache is turned on but costs are not reduced

There are cases where prompt caching is applied but costs and delays are hardly reduced. This is usually because the repeated prefix is ​​short, the beginning of the system prompt changes for each request, or the search document is inserted into the beginning of the prompt. Even if the provider supports cache, the effect will be small if the prompt layout does not fit the cache.

Another failure is to only look at the average token. LLM fees vary greatly with p95 and p99 requests. Most requests are short, but if some requests include 20 long documents and generate long responses, the monthly cost estimate will be wrong. The token budget is not an average optimization, but rather an operational policy that sets per-request upper limits and exception handling.

Implementation example: Dividing the prompt layout into cache-friendly

Copy1[stable prefix]2- product role3- response policy4- tool contract5- safety and citation rules6 7[semi-stable]8- tenant policy version9- prompt template version10 11[dynamic suffix]12- user question13- retrieved chunks14- conversation summary

Put static policies at the front, and questions and search results at the back. When the template version changes, the decrease in cache hits is considered a normal event and recorded in the metrics. If the dynamic suffix exceeds the budget, the retrieval top-k should be reduced or replaced with a summary context.

Checklist application results

Having this metric allows you to say the cause, not like "the prompts are long and that's expensive", but something like "the cached input rate dropped in version 7 and the retrieval suffix pushed up p95". Cost optimization is more about measurable budget management than it is about statement sense.

Example of failure response: Budget exceeded for long document question

Let's say a user uploads an 80-page policy document and asks, "Summary everything and find exceptions." A good backend doesn't immediately put the entire document into the prompt. First, create a budget plan by looking at the document length, number of chunks, and question types.

In operations, budgets are set differently for each question type. For simple definition questions, keep top-k small and output reserve small. Comparative analysis or exception clause exploration broadens the retrieval candidates, but creates intermediate summaries and includes only source chunks in the final answer. In this case, prompt caching is helpful for fixed policies and schemas, but may not have much effect on the upload document itself.

A borderline case is a retry storm. When a rate limit error occurs, retrying the same long prompt multiple times increases both cost and delay. You should set a separate retry budget, and if it exceeds the budget, you should inform the user that “the documents will be processed separately” or “a narrower question is needed.” token budget is not a simple optimization, but a device that turns failure into a product experience.

10. Q&A

Q1. Does prompt caching alone solve the cost problem?

no. prompt caching is advantageous for repetitive prefixes. Areas that change every time, such as RAG context or user questions, require improvement in token budget and retrieval quality.

Q2. Is it okay to use the system prompt for a long time?

If it is repeated and cached well, the cost burden can be reduced. However, long system prompts create maintenance and quality regression risks, so versioning and eval are necessary.

Q3. Are output tokens also cached?

prompt caching is primarily concerned with input prefix reuse. The output length must be managed separately with max token and response policy.

11. References and uncertainty

References

• OpenAI prompt caching:https://platform.openai.com/docs/guides/prompt-caching
• OpenAI Rate Limits:https://platform.openai.com/docs/guides/rate-limits
• OpenAI Cookbook — prompt caching:https://developers.openai.com/cookbook/examples/prompt_caching_201

uncertainty

• Cache application terms, support models, and pricing may vary from time to time.
• Token estimation may differ from actual provider usage.

---