An Agent Loop Is a State Machine, Not a Model Call

Key takeaways

• Agent loop is a repeatable state machine rather than a one-time model API call.
• The model can create a tool request while generating text, and the runtime injects the result back into a message.
• Stream events, usage updates, stop reasons, and tool calls must all be handled as runtime events.
• The continuation condition must be clear to prevent infinite loops, missing tool results, and runaway costs.

In this article, we cover the query loop, the heart of agent runtime.

Many LLM apps think of model calls as “one request, one response.” But agents are different. The model may request the use of a tool in the middle of a response, and the runtime must execute the tool and send the result back to the message to allow the model to continue making inferences.

In other words, the agent loop is not a model call function but a state machine that repeats streaming events and tool result injection.

1. Why an agent cannot be created with just one model call

Once the model requests a tool, the response is not over. Rather, agent execution begins from then on.

Copy1# How to read: The items below are summarized examples to quickly check the action flow.2messages3→ model stream4to assistant text Some output5to tool request occurs6run to tool runtime7Create to tool result message8Reinject to messages9to model stream resume10→ final answer

If you fail to handle this iteration, your tool calls will just be one-off function executions. The model cannot make its next decision based on observations of the external world.

2. Query loop status

The agent loop manages the following states:

These states do not end with a single provider call. It must be continued when inserting the tool result and calling the model again.

3. Convert stream event to runtime event

If the event name given by the provider API is spread throughout the app, it is difficult to change the provider. Therefore, the query loop must convert the provider stream into an internal runtime event.

This way, the session shell, ledger, and cost tracker do not need to know the provider SDK type.

4. Tool result reinjection

The tool result is not a simple output string. Observations that will be used in the next model call.

Even if the tool fails, you must create a result message. If the model does not have a corresponding result for the requested tool call, the next inference may become unstable.

Copy1# How to read: The items below are summarized examples to quickly check the action flow.2tool request id: T-173→ schema validation failed4→ permission denied5→ execution failed6→ success result

In all four cases above, it is a “result”. In addition to viewing success as an outcome, failure must also be viewed as an observation that the model can read.

Practical Tips Instead of treating tool errors only as Python exceptions, try changing them to model-visible error results. The agent can choose different strategies on its own.

5. Continuation condition design

It must be clear when the agent loop continues running and when it stops.

If these conditions are scattered, the agent may run infinitely, tool results may be missing, or costs may be greater than expected.

6. Agent loop viewed through concept code

The code below is newly written code for explanation purposes.

Copy1# How to read: Conceptual code that describes runtime boundaries, not a clone of the actual implementation.2class LoopState:3    # Initializes runtime dependencies and state storage that the object will reference in later steps.4    def __init__(self, messages, tool_context):5        self.messages = list(messages)6        self.tool_context = tool_context7        self.turn_count = 08        self.usage = UsageSnapshot()9        self.cancelled = False10 11# It reads the model stream, executes a tool request, and injects the results back into a model message.12async def run_agent_loop(initial_messages, runtime):13    state = LoopState(initial_messages, runtime.tool_context)14 15    while True:16        if state.turn_count >= runtime.limits.max_model_rounds:17            await runtime.ledger.write("turn_limit_reached", state.turn_count)18            return FinalAnswer(reason="turn_limit")19 20        if runtime.abort_signal.requested:21            return FinalAnswer(reason="cancelled")22 23        request = runtime.provider_adapter.build_request(state.messages, state.tool_context)24        tool_requests = []25 26        async for event in runtime.provider_adapter.stream(request):27            if event.type == "assistant_text_delta":28                runtime.shell.publish(event)29 30            elif event.type == "tool_request_ready":31                tool_requests.append(event.tool_request)32 33            elif event.type == "usage_updated":34                state.usage.merge(event.usage)35                runtime.accounting.update(state.usage)36 37        if not tool_requests:38            return FinalAnswer(reason="assistant_done", usage=state.usage)39 40        async for result in runtime.tools.execute_batch(tool_requests, state.tool_context):41            state.messages.append(result.to_model_message())42            runtime.shell.publish(result.to_screen_event())43            await runtime.ledger.write("tool_result", result.summary())44 45        state.turn_count += 1

In this architecture, the provider adapter and tool runtime do not know each other's internal implementation. A query loop connects the two.

7. AI utilization developer perspective

When using an AI coding agent, it is good to observe how long tasks progress.

• Does the screen indicate that the model has requested the tool?
• Do you see a summary of the results of running the tool?
• Does the agent try a different strategy for a failed tool?
• Are there limits when iterative execution becomes too long?
• Are costs/tokens updated mid-term?

An agent without this information being visible may seem convenient, but it makes auditing and debugging difficult in a team environment.

8. Agent developer checklist

Copy1# How to read: The items below are summarized examples to quickly check the action flow.2Query Loop Checklist3 4[ ] The model call is designed as a repeatable loop.5[ ] The provider stream is converted to an internal runtime event.6[ ] Tool requests are collected and then passed to the tool runtime.7[ ] The tool result is reinjected as a model-visible message.8[ ] A failed tool request also creates a result message.9[ ] The conditions for max turn, cancel, budget, and context compaction are clear.10[ ] usage/cost is also updated in mid-stream events.

Failure case: Moving to the next call without entering the tool result

One of the most dangerous bugs in the agent loop is when the model issues a tool request, but the runtime does not correctly put the result back into the message history. On the screen it looks like the tool ran, but the next model call is missing the results. The model then requests the same tool again, responds by imagining the outcome, or continues an operation that has already failed as if it had succeeded.

This issue occurs more often with streaming UI. The screen moves based on the token stream, the backend moves based on tool events, and the ledger can store only the final message. If the three layers do not share the same turn ID and tool call ID, a ghost state will occur that is difficult to reproduce. The user says “I just read the file,” but the model replies that it hasn’t read it, and so on.

State transition example

This failure can be reduced by having the query loop as a state machine.

Copy1idle2-> streaming_model3-> collecting_tool_requests4-> executing_tools5-> injecting_tool_results6-> streaming_model7-> done

For each state, the events that can enter and the events that can exit must be determined. To start a new model stream inexecuting_toolsstate, you must check whether all required tool results have been entered into history. Conversely, optional telemetry events do not need to be included in model history. Without this distinction, the context becomes unnecessarily large and important results are missed.

Checklist application results

When reviewing an actual implementation, look at “where the repetition conditions are” rather than “how many times do you call the model?” max turn, max tool calls, cancel signal, budget exceeded, and provider stop reason must all be determined in the same loop controller. If each adapter retries randomly, the agent becomes difficult to stop. Especially for products with high costs, even if the usage update arrives late, the budget must be recalculated before the next turn.

Implementation comparison: simple while loop and operational loop controller

When first creating an Agent loop, thewhile tool_calls:format seems sufficient. However, in production, conditions such as outage, cost, tool failure, context compression, and user approval are introduced. A simple while loop would easily spread these conditions to each handler.

For example, let's say a file search tool fails. In the operational loop, the failure reason is put intool_resultand passed back to the model. The model may try a different path, or it may reply to the user that "current file cannot be read". In a simple loop, an exception may pop out and only a generic error may remain on the screen.

There are also borderline cases. What should I do if there are more than two tool requests, one is successful and one is permission denied? All results must be collected and put into observations of the same turn. If only successful results are entered, the model may mistakenly think that a rejected operation is successful, and if only rejections are entered, the information already obtained is discarded. The loop controller should be a layer that checks whether the history is complete before calling the next model.

finish

Understanding the agent loop reveals the essence of an AI agent. The model does not run alone. The runtime reads the stream, executes the tool, puts the results back, and decides when to stop.

In the next article, we will look at the provider API boundary. How should we design model call boundaries so that our product logic is not tied to the provider SDK type?