Where to draw the line between LLM and internal API
In this article, we outline what boundaries should be drawn when connecting LLM and internal API using function calling.
function calling allows models to interact with external systems. In addition to answering user questions, you can also query data in the database, create tickets, search documents, and run workflows.
But an important question arises here.
How much should we leave to the model? Which API will be exposed as a tool? Who will do the permission checking? How do you bounce back when you fail?
Analysis base date: 2026-05-12 Practice environment: OpenAI API, FastAPI, PostgreSQL Key references: OpenAI function calling Docs, Structured Outputs Docs, AWS Idempotency
Key takeaways
- function calling should be understood as a method in which the model structures the call intent and delivers it to the backend, rather than directly executing an internal API.
- The backend tool executor must have actual execution permission.
- Read-only tools and mutating tools must be separated.
- State change tools may require idempotency, audit logs, and human approval.
- The tool schema should not expose the entire internal API, but only provide the minimum interface required for the model.
1. How to understand function calling
function calling is dangerous when understood as the ability of a model to “directly execute” a function. More precisely, it is a function that suggests in a structured form which function the model should call with which arguments.
The actual execution must be done by the backend.
# This is an example.User request→ LLM decides tool call→ Backend validates tool call→ Permission check→ Execute internal API→ Return tool result to model→ Generate final responseThe model is the decision assistant, and the backend is the executor.
2. What is Tool Boundary?
Tool Boundary is the boundary of functions exposed to the model.
Bad example:
# This is an example.execute_sql(query: string)This tool is so powerful. Models can generate arbitrary SQL, and permissions and safety are difficult to control.
Good example:
# This is an example.search_documents(query: string, scope: enum, limit: integer)get_document_summary(document_id: string)create_support_ticket(title: string, description: string, priority: enum)The actions required of the model should be limited to small tools.
3. Separation of Read Tool and Write Tool
| Tool type | example | risk | protection device |
|---|---|---|---|
| Read | Document search, status inquiry | low to medium | Permission filter, rate limit |
| Write | Create ticket, send email | medium to high | idempotency, approval |
| Destructive | Delete, refund, change permissions | height | Default ban or strong approval |
For initial function calling, it is safe to start with a read-only tool.
4. Tool schema design
Example document search tool:
// This is an example JSON structure.{ "name": "search_documents", "description": "Search for relevant content in documents accessible to the user.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "Natural language query to search" }, "scope": { "type": "string", "enum": ["official_docs", "papers", "tech_blogs"] }, "limit": { "type": "integer", "minimum": 1, "maximum": 10 } }, "required": ["query", "scope", "limit"], "additionalProperties": false }}The design points are as follows:
# This is an example.[ ] Limit the scope with enum.[ ] limit Sets the maximum value.[ ] Internal ID or permission conditions are determined by the server.[ ] The user's original text input is not passed on to the internal API as is.5. Authorization and authentication
Permission checking should be done by the server, not the model.
# This is an example.User A asks a questionto model suggests calling search_documentsto server checks user A's document permissionsto Search only authorized scopesto return resultIt is not enough to tell the model, “Retrieve only documents accessible to this user.” The actual filter must be enforced on the backend.
6. Pre-execution Verification and Human Approval
State change tools require verification before execution.
# This is an example.Create LLM tool call→ schema validation→ business rule validation→ permission check→ risk classification→ human approval if needed→ executeFor example, sending emails, creating tickets, or changing customer status may require human approval.
// This is an example JSON structure.{ "tool": "create_support_ticket", "arguments": { "title": "Redis cache issue", "description": "Users reported Redis cache miss issues.", "priority": "medium" }, "requires_approval": true}7. Audit logs and reproducibility
function calling must leave an audit log.
| log entry | reason |
|---|---|
| user_id | who requested it |
| tool_name | Which tool was called |
| tool_arguments | What arguments were used |
| validation_result | Verification passed or not |
| execution_result | execution result |
| trace_id | Failure analysis |
| prompt_version | Reproducibility |
| model | quality analysis |
However, sensitive information must be masked.
8. Failure and rollback
Tool execution may fail.
| failure type | react |
|---|---|
| schema validation failed | Retry model or fallback |
| No permission | refuse to run tool |
| Internal API timeout | retry or user guidance |
| duplicate request | Return existing results with idempotency key |
| Invalid state change | rollback or compensation transaction |
It is safer not to expose state change tools unless there is a rollback plan.
9. Practical checklist
# This is an example.[ ] Are the read tool and write tool separated?[ ] Does the tool schema follow the principle of least privilege?[ ] Are there input restrictions such as enum, max limit, etc.?[ ] Is permission checking enforced on the server?[ ] Does the status change tool have an idempotency key?[ ] Is there human approval for hazardous work?[ ] Are all tool calls left in the audit log?[ ] Is there an explainable response to the user in case of failure?Failure case: When the internal API is exposed as a tool
A common mistake seen in function calling is exposing an already existing backend API almost as a tool. For example, if the model can call endpoints such asPOST /tickets,PATCH /users/:id, andDELETE /documents/:id, the implementation will be faster. However, these APIs are designed with the assumption that they are explicitly called by a person or server. If you give it to the model as is, the scope of arguments, permissions, approval, and retry implications are too broad.
In particular, the write API is dangerous without idempotency. If the model sees a timeout and tries the same tool call again, but the first request was actually successful, two tickets may be created. For tasks that require confirmation from the user, simply writing “call with caution” in the tool description is not enough. The server must check the approval state before final execution.
Implementation example: Creating a command API for a model
If you have a narrow command surrounding the internal API, the boundary becomes clearer.
type CreateTicketCommand = { kind: "create_support_ticket"; idempotencyKey: string; title: string; summary: string; priority: "low" | "normal" | "high"; requiresHumanApproval: true;};This command is intentionally smaller than the internal ticket API. Values such as assignee, internal label, and billing field are filled in by the server, making the model uneven. The model structures the proposal, “I need to create a ticket,” and the backend checks permissions and approval and calls the internal API.
Separate design interpretation from formal facts
What the official documentation provides is that “the model can make structured tool calls.” However, “which internal APIs will be exposed as tools,” “which tasks will be sent after approval,” and “which fields will be calculated by the server” are product design interpretations. If you mix the two, it's easy to mistake tool support for operational safety.
| decision | official function | Product Design Responsibility |
|---|---|---|
| schema | Specify tool input format | Field minimization and versioning |
| execution | Receive tool call | authority, authorization, idempotency |
| error | Pass failure response | Retry availability and user phrase |
| audit | Call information can be obtained | Store who did what, why, and what |
With this separation, function calling can be treated as a backend boundary rather than a model feature.
Practical example: Creating a customer support ticket
Let's say your customer support chatbot needs to view the conversation and create a ticket. A bad design would be to have the model populate every field in the internal ticket API. A good design is for the model to structure only the “ticket creation intent” and the server to populate the user information, organization, SLA, responsible queue, and approval status.
| field | Can the model fill it? | reason |
|---|---|---|
| title | yes | Can be extracted from conversation summary |
| summary | yes | Corresponds to user problem description |
| priority | limitedly | Enum and server calibration required |
| assignee | no | Organizational policies and work status required. |
| customerTier | no | Internal CRM permission required |
| idempotencyKey | Create server | Avoid duplicate creation |
Failure case is retry after timeout. The model calledcreate_support_ticketand the provider response was delayed. The user makes the request again, and the model calls the same thing one more time. If you do not have an idempotency key, you will have two tickets. If there is no approval status, the user may say "just create a draft" but an actual ticket will be sent.
Therefore, it is safer to divide the write tool into three steps. First, the model creates a command draft, second, the server attaches validation and idempotency keys, and third, the user or policy approves and executes the internal API. In this structure, the result of function calling is not a side effect, but an intermediate product that turns side effect candidates into verifiable commands.
10. Q&A
Q1. Can I expose the internal API as a tool?
Not recommended. Tools for models should have a smaller and more limited interface than the internal API.
Q2. What if I select the wrong tool for the model?
The server must verify and reject the tool call. Model judgments are suggestions only.
Q3. Is function calling the same as Agent?
They can be components of an Agent, but they are not the same. function calling is an interface that structures external function calls.
11. References and uncertainty
References
- OpenAI function calling:https://platform.openai.com/docs/guides/function-calling
- OpenAI Structured Outputs:https://platform.openai.com/docs/guides/structured-outputs
- AWS Builders Library — Idempotent APIs:https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
uncertainty
- function calling's detailed API and support model may vary depending on the time.
- Actual tool approval criteria must be aligned with the organization's security and operational policies.
댓글
GitHub 계정으로 로그인하면 댓글을 남길 수 있습니다. 댓글은 GitHub Discussions를 통해 운영됩니다.