Separating Provider APIs from Product Logic

Key takeaways

• Provider API boundary is not a simple SDK wrapper.
• Translates internal messages, tool schema, system context, and model options into provider payload.
• The streaming response must be converted to an internal runtime event.
• If the provider SDK type leaks throughout the session shell, tool runtime, and ledger, maintenance becomes difficult.

In this article, we will look at the model provider API boundary.

The agent runtime naturally depends on the model API. However, the entire product should not be caught up in the type and event name of the provider SDK. This is because the entire app shakes when you change the provider, add a fallback, or change the streaming processing method.

If you look at Claude Code-type agents from a runtime perspective, the model call boundary is not a simple wrapper. This layer translates between internal messages and provider payloads, selects tool schemas, converts streaming responses into runtime events, and tracks usage and cost.

1. Problems that occur when the Provider SDK type leaks throughout the app

At first, it is convenient to use the provider SDK type as is. However, this becomes a problem as the agent grows.

• The session shell needs to know the provider event name.
• ledger must read the provider usage format directly.
• The tool runtime is bound to the provider tool schema format.
• To add a fallback model, all layers must be modified.
• It is difficult to mock the provider stream in tests.

Therefore, the internal runtime must have its own event vocabulary.

Copy1# How to read: The items below are summarized examples to quickly check the action flow.2provider stream event3→ provider adapter4→ runtime event5→ shell / ledger / accounting / tool loop

2. API boundary responsibility

With this layer, even if the model provider changes, other layers of the agent runtime are likely to remain the same.

3. Message encoding and tool schema selection

It is not cost-effective to always send all tool schemas to the model. Conversely, if the required tools are missing, the model will not be able to make appropriate requests.

Therefore, the API boundary must have a policy for selecting the tools needed for the current turn.

Tool schema selection is connected to the tool runtime, but it is cleaner to do the final assembly into the provider payload at the API boundary.

4. Stream event translation

The stream event structure may be different for each provider. For internal runtime, you only need to know the following common events.

This way, the session shell only needs to know that it is “displaying partial text” and does not need to know what raw events the provider gave.

5. Usage and cost accounting

In streaming API, usage information may not come only at the end. It may be updated as an intermediate event, and detailed items such as cache token or reasoning token may be expressed differently for each provider.

Therefore, it is better to have a structure where usage continues to be absorbed in the middle of the stream rather than being read once from the final response.

caution: In a long-running agent, if cost tracking is calculated only at the end, errors may increase in situations of intermediate cancellation, retry, fallback, and compaction. If budget enforcement is necessary, mid-stream usage updates must be processed.

6. Provider adapter viewed through concept code

The code below is newly written code for explanation purposes.

Copy1# How to read: Conceptual code that describes runtime boundaries, not a clone of the actual implementation.2class ProviderAdapter:3    # Initializes runtime dependencies and state storage that the object will reference in later steps.4    def __init__(self, client, encoder, pricing):5        self.client = client6        self.encoder = encoder7        self.pricing = pricing8 9    # Encodes the runtime message and tool context into a request payload that the provider understands.10    def build_request(self, messages, tool_context):11        return {12            "messages": self.encoder.encode_messages(messages),13            "tools": self.encoder.encode_tools(select_tools(tool_context)),14            "system": self.encoder.encode_system(tool_context.system_context),15            "options": self.encoder.encode_options(tool_context.model_options),16        }17 18    # Converts raw events into runtime events and emits them while traversing the provider stream.19    async def stream(self, request):20        async for raw in self.client.open_stream(request):21            event = self.translate(raw)22            if event is not None:23                yield event24 25    # Map event types for each provider to the standard RuntimeEvent used inside the product.26    def translate(self, raw):27        if raw.kind == "text_piece":28            return RuntimeEvent("assistant_text_delta", text=raw.value)29 30        if raw.kind == "tool_call_done":31            return RuntimeEvent("tool_request_ready", tool_request=decode_tool_request(raw))32 33        if raw.kind == "usage":34            normalized = normalize_usage(raw.usage)35            return RuntimeEvent("usage_updated", usage=normalized)36 37        if raw.kind == "done":38            return RuntimeEvent("assistant_finished", reason=raw.reason)

Here, the provider adapter only exports internal runtime events. If this contract is stable, provider replacement becomes easy.

7. AI utilization developer perspective

AI tool users do not see provider boundaries directly. But you can feel the quality by the following phenomenon:

• Are the UI and tool execution flow stable even when the model changes?
• Is cost information updated mid-stream?
• Are there too many tool schemas and the model is picking the wrong tools?
• Provider When an error occurs, is an understandable message displayed to the user?
• Even with fallbacks, are transcripts and cost records consistent?

8. Agent developer checklist

Copy1# How to read: The items below are summarized examples to quickly check the action flow.2Provider API Boundary Checklist3 4[ ] The provider SDK type does not leak to the session shell and tool runtime.5[ ] There is a single assembly point that turns the internal message into a provider payload.6[ ] The tool schema selection policy is specified.7[ ] Stream event is converted to internal RuntimeEvent.8[ ] usage/cost is updated even in the middle of the stream.9[ ] Retry/fallback/non-streaming path maintains the same event contract.10[ ] Feature differences for each provider are isolated inside the adapter.

Failure case: When the provider SDK type spreads to the entire product

Initially, it is faster to use the message type of the provider SDK as is. Sincemessages,tools,stream, andusagealready exist, there seems to be no need to create a separate internal type. However, if you add a second provider, operate streaming and non-streaming fallback together, or select the tool schema differently for each workspace, the SDK type binds the entire product logic.

For example, if the UI directly knows the provider's chunk event name, the moment you change the provider, the UI will also change. If ledger directly stores the usage field for each provider, cost reports will have different meanings for each provider. If the tool runtime directly creates the provider tool schema, the same tool will have different validation rules for each provider. Ultimately, the entire product, not just the adapter, becomes provider integration.

Comparison table: thin wrapper and real boundary

This difference becomes apparent during operation. When a provider timeout occurs, if there is a boundary, retry policies, user messages, and ledger records can be processed in the same way. All you need is a wrapper, and each screen and service function interprets provider errors independently.

Implementation example: internal event contract

Copy1type RuntimeEvent =2  | { type: "assistant_text_delta"; turnId: string; text: string }3  | { type: "tool_request"; turnId: string; callId: string; name: string; input: unknown }4  | { type: "usage"; turnId: string; inputTokens: number; outputTokens: number; cachedInputTokens?: number }5  | { type: "provider_error"; turnId: string; retryable: boolean; message: string };

Provider adapter converts external formats such as OpenAI, Anthropic, and local model into this contract. The remaining products do not know the chunk name for each provider. This allows the UI to display streaming deltas, the ledger to store usage, and the tool runtime to focus on processing requests. Replacing providers is still difficult, but the blast radius is reduced to within the adapter boundaries.

Failure response example: switching from streaming provider to non-streaming fallback

The value of the provider boundary is revealed when there is a failure. If the product logic is tied to a raw SDK event when a streaming call is interrupted, the fallback implementation is scattered to the screen, ledger, and cost calculation, respectively. On the other hand, if there is an internal event contract, you can exportRuntimeEvent, which also has the same fallback.

For example, let's say that a tool call delta from an Anthropic stream comes in several pieces, and another provider only gives completed tool calls. The entire product does not need to know this difference. When the adapter internally puts together the pieces and creates atool_request_readyevent, the query loop calls the tool runtime in the same way. The reason this comparison is important is because provider replacement becomes a failure response strategy rather than a simple cost optimization.

A boundary case is when the fallback provider does not support the same functionality. For example, the original provider may support a specific response schema, but the fallback provider may not. At this time, the adapter should specify a capability mismatch rather than quietly sending another request. Then, the product can choose between “possible fallback”, “inform user”, or “stop operation” as its policy.

finish

Although the Provider API boundary is not visible, it greatly influences agent productivity. This boundary must be thin and clear to enable reliable provider changes, fallbacks, cost tracking, and tool schema optimization.

In the next article, we will look at tool runtime. You should not run a tool immediately after the model requests it. Registry, schema validation, permission, and orchestration are required.