Production AI agents need controls for API cost and response latency.
Sending every request to the same large model can increase cost as traffic grows. Smaller models may reduce per-token costs, but they must still meet the accuracy and tool-use requirements of the task. Compare candidates using the actual workload and current provider prices.
This post distinguishes Dynamic Routing (choosing a tier before execution) from Cascade (escalating after a failed validation). The roughly 70% saving described below is a hypothetical cost calculation, not a production result.
1. Single Model Bottleneck & The Need for Dynamic Routing
Assume a workload with three complexity levels. The percentages illustrate the calculation; they were not measured in a real service:
- Simple Lookup & Classification (assumed 50%): "Check my order delivery status"
- Conditional Extraction & Tool Calling (assumed 35%): "Retrieve transactions over $100 from last month and format as CSV"
- Complex Multi-step Reasoning & Coding (assumed 15%): "Analyze backend error logs to locate memory leaks and generate a PR diff"
These assumptions create 85% candidate traffic for cheaper tiers, not an 85% guarantee that cheaper models will solve the tasks. Accuracy, retry frequency, and tail latency require separate measurements.
Dynamic Routing uses a lightweight Router Classifier to assign requests to a suitable model tier.
The model names in the illustration are examples of routing tiers. Compare candidates using current provider catalogs and prices before building a routing policy.
2. Dynamic Routing & Cascade Architecture Patterns
A. Intent/Complexity-based Classifier Routing
A lightweight classifier routes incoming requests directly to the appropriate tier based on complexity and required tool signatures.
User request → Router classifier → Small / Medium / Large model (rules or smaller LLM)B. Fallback Cascade (Sequential Escalation)
Attempts execution with a smaller model first. If schema checks, tool results, or task-specific validation fail, it escalates to a stronger model. A model-generated confidence number alone is not proof of correctness.
3. Tier Classification Example (Not a Production Dispatcher)
from enum import Enumfrom pydantic import BaseModel, Fieldfrom openai import OpenAI class ModelTier(str, Enum): FAST = "fast" BALANCED = "balanced" ADVANCED = "advanced" class RouteDecision(BaseModel): selected_tier: ModelTier = Field(description="Selected model tier") complexity_score: float = Field(ge=0, le=1, description="Estimated complexity, 0.0 to 1.0") reasoning: str = Field(description="Rationale for routing decision") class DynamicModelRouter: def __init__(self, client: OpenAI, classifier_model: str): self.client = client self.classifier_model = classifier_model def route_query(self, user_prompt: str, available_tools: list[str]) -> RouteDecision: router_system_prompt = ( "You are a Request Router for an AI Agent workflow. " "Analyze user prompt and available tools to select the optimal model tier." ) completion = self.client.beta.chat.completions.parse( model=self.classifier_model, messages=[ {"role": "system", "content": router_system_prompt}, {"role": "user", "content": f"Query: {user_prompt}\nTools: {available_tools}"} ], response_format=RouteDecision ) parsed = completion.choices[0].message.parsed if parsed is None: raise RuntimeError("Structured classification result was missing") return parsed4. Hypothetical Cost Calculation — Not Production Results
Evidence required before claiming savings: Compare the router plus all downstream model and retry costs with the same baseline workload. For each task class, log its routing decision, success criterion, retry count, actual billed tokens and end-to-end latency. A lower average model price does not prove total cost savings if escalation or wrong answers trigger expensive repeats. The numerical result below is a scenario calculation, not a measurement of a deployed service.
For identical workloads, the estimated saving is defined by the same baseline, including classifier and retry overhead:
Baseline cost = sum(request input tokens × baseline input rate + request output tokens × baseline output rate)Routed cost = sum(selected model input/output cost) + classifier cost + failed-request retry costSaving (%) = (Baseline cost - Routed cost) / Baseline cost × 100Assume the request mix is 50% simple, 35% medium, and 15% complex. Assume the same input/output tokens for each request, a small-tier unit cost of 1/15 and medium-tier cost of 1/5 of the large-tier cost.
- All large-model requests: normalized cost = 1.000.
- Routed model calls, before classification and retries: 0.50/15 + 0.35/5 + 0.15 = 0.2533.
- Classifier and escalation cost (assumed 5% of baseline): 0.0500.
- Total hypothetical cost: approximately 0.3033, or 69.7% saved.
These are explicit assumptions, not provider invoices, benchmarks, or measurements. More tokens per complex request, additional router calls, and frequent retries can reduce or eliminate the saving.
For a real evaluation, record input and output tokens, rates, classifier overhead, escalation frequency, and task pass/failure rates. Compare P50/P95/P99 latency on the same test set. The Python example only classifies a tier; it does not invoke the selected provider or implement escalation.
Conclusion
Dynamic Routing selects a candidate tier; Cascade retries after validation fails. The approximately 70% saving here is an arithmetic example, not measured performance. Compare cache hit rate, token budgets, classifier accuracy, retry cost, and task success before adopting either pattern.
댓글
GitHub 계정으로 로그인하면 댓글을 남길 수 있습니다. 댓글은 GitHub Discussions를 통해 운영됩니다.