Structured Outputs in Practice

Create model output as a JSON Schema contract

In this article, we summarize how to use Structured Outputs to handle LLM responses as JSON Schema-based contracts.

The first question you encounter when attaching an LLM response to a service is not “Is the answer plausible?” “Can the backend parse it?”

If your model answers in free text, your frontend, DB, and subsequent APIs won't be able to use it reliably. In particular, tasks that require structured results, such as document summarization, information extraction, classification, routing, and tool call preprocessing, require model outputs to be contracts.

Analysis base date: 2026-05-12 Practice standard environment: OpenAI API, Python, Pydantic, FastAPI Main reference materials: OpenAI Structured Outputs Docs, JSON Schema, Pydantic

Key takeaways

• Structured Outputs is a function that makes model responses follow the specified JSON Schema.
• LLM responses must be treated as contracts like API responses within the service.
• Without a schema version, it is difficult to track changes in prompts and response structures.
• Validation failure should be viewed as a normal operating scenario and a retry/fallback should be designed.
• Instead of leaving all fields to the model, the fields to be calculated by the server should be filled in by the server.

1. Why Structured Outputs?

Free text responses are great for humans to read, but difficult for systems to process.

For example, let's say you made the following request to the model:

Copy1# This is an example.2Summarize the main content of this document and include sources and credibility.

The model can answer in many different ways.

Copy1# This is an example.2The key point is that Redis cache reduces DB load. The source is the Redis official document.

or:

Copy1# This is an example.2{3"summary": "Redis cache reduces DB load",4"source": "Redis Docs"5}

Services hate this volatility. The frontend expects an array ofcitations, but if the model gives it the stringsource, it breaks.

2. Difference between JSON mode and Structured Outputs

JSON mode helps you generate valid JSON. However, this is different from necessarily adhering to a specific schema.

Structured Outputs focuses on ensuring that the response follows the JSON Schema defined by the developer.

In practice, schema-based output is safer than JSON mode.

3. Response schema design principles

A good schema doesn't leave everything to the model.

4. Document Q&A response schema example

Copy1// This is an example JSON structure.2{3  "type": "object",4  "properties": {5    "answer": {6      "type": "string",7      "description": "Answers to user questions"8    },9    "confidence": {10      "type": "string",11      "enum": ["high", "medium", "low"]12    },13    "citations": {14      "type": "array",15      "items": {16        "type": "object",17        "properties": {18          "document_id": { "type": "string" },19          "chunk_id": { "type": "string" },20          "quote": { "type": "string" }21        },22        "required": ["document_id", "chunk_id", "quote"]23      }24    },25    "needs_human_review": {26      "type": "boolean"27    }28  },29  "required": ["answer", "confidence", "citations", "needs_human_review"],30  "additionalProperties": false31}

This schema clearly separates answers, their credibility, source, and whether they require review.

5. Validation and fallback

Even when using Structured Outputs, backend validation is required. Output from external systems should always be verified.

Copy1# This is example code.2from pydantic import BaseModel3from typing import Literal4 5# This declaration shows an example flow.6class Citation(BaseModel):7    document_id: str8    chunk_id: str9    quote: str10 11# This declaration shows an example flow.12class AnswerOutput(BaseModel):13    answer: str14    confidence: Literal["high", "medium", "low"]15    citations: list[Citation]16    needs_human_review: bool

Failure Scenario:

Copy1# This is an example.21. Model response schema validation failed32. Retry once43. If it still fails, return a fallback response54. Save failure cases in eval/debug dataset

Fallback example:

Copy1// This is an example JSON structure.2{3  "answer": "Could not reliably generate answers. Please be a little more specific with your question.",4  "confidence": "low",5  "citations": [],6  "needs_human_review": true7}

6. Schema versioning

Response structures change over time. Therefore, schema version is required.

Copy1# This is an example.2answer.v13answer.v24classification.v15extraction.receipt.v1

The schema version must be recorded along with the prompt version.

7. Separate DB storage and API response

There is no need to make the model output schema and API response schema exactly the same.

Model output:

Copy1// This is an example JSON structure.2{3  "answer": "...",4  "confidence": "high",5  "citations": []6}

API response:

Copy1// This is an example JSON structure.2{3  "answer_id": "ans_123",4  "answer": "...",5  "confidence": "high",6  "citations": [],7  "usage": {8    "input_tokens": 1000,9    "output_tokens": 30010  },11  "trace_id": "0af..."12}

usage,trace_id, andanswer_idare values ​​added by the server.

8. Practice risks

9. Practical checklist

Copy1# This is an example.2[ ] Did you distinguish between model output schema and API response schema?3[ ] Are the required fields clear?4[ ] Have you restricted additionalProperties?5[ ] Are enum values ​​connected to subsequent logic?6[ ] Do you save schema_version?7[ ] Is the validation failure rate recorded as a metric?8[ ] Is there a retry/fallback policy in case of failure?9[ ] Did you prevent the model from randomly generating citations?

Failure case: JSON received but no contract

Using Structured Outputs does not immediately create a stable API contract. The most common failure is when you create a schema but don't have a version, validation, or fallback policy. The model returns JSON, which the parser also passes, but the meaning may not be what the downstream service expects. For example, if it is not documented what thresholdrisk: "medium"means, the UI, DB, and alert rule interpret it differently.

Another problem is leaving all the fields to the model. Some values, such ascreatedAt,sourceId,confidenceScore, andneedsReview, must be calculated by the server. If you trust the timestamp or internal ID filled in by the model, data consistency will be broken. Structured Outputs is a tool for controlling output format, not a replacement for business rule validation.

Example implementation: schema version and server calculated fields

Copy1type ExtractionV1 = {2  schemaVersion: "customer_intent.v1";3  intent: "refund" | "upgrade" | "bug_report" | "unknown";4  evidence: string[];5  modelConfidence: number;6};7 8type StoredExtraction = ExtractionV1 & {9  requestId: string;10  validatedAt: string;11  needsReview: boolean;12};

The model only producesExtractionV1. The server calculatesrequestId,validatedAt,needsReview. In particular,needsReviewmust not only look at model confidence, but also look at server information such as user rating, sensitive keywords, and previous failure history.

Comparison table: output formats and operating agreements

Based on this table, Structured Outputs are at the front of the validation layer. The backend still needs to validate external input and check for business invariants before saving. This ensures that model replacement or prompt changes do not directly lead to downstream failures.

Practical application example: storing inquiry classification results

Let's say you want to create a function that classifies customer inquiries into intents such as refund, billing, bug, and account. It is best to keep the model output schema small.

Copy1{2  "schemaVersion": "ticket_intent.v1",3  "intent": "billing",4  "evidence": ["invoice number", "charged twice"],5  "confidence": "medium"6}

Here, the server separately addsrequestId,receivedAt,customerId, andneedsHumanReview. In particular,needsHumanReviewis not determined by model confidence alone. Sensitive intents such as refunds, personal information, and payment failures can be sent for review even if the confidence level is high.

A failure is to make your schema too large too quickly. If you receivedepartment,assignee,refundAmount,legalRisk, andreplyDraftat once from the beginning, it is easy to mix up the business meaning even if the model format matches. Separating the classification schema and the answer draft schema reduces the scope for failure. Even if the classification is incorrect, answer generation may not start, and even if the answer draft fails, a classification log may be left.

The comparison criteria are simple. What Structured Outputs guarantees is that “the response is close to the schema shape.” What the service must guarantee is that “the value matches the current user, permissions, DB state, and policy.” Separating the two prevents you from running the wrong subsequent API just because you received JSON.

10. Q&A

Q1. If I use Structured Outputs, can I avoid using validation code?

no. Validation still needs to be done in the backend. It is fundamental not to trust external system results.

Q2. Should all LLM responses be JSON?

no. The final answer you show your users might be Markdown. However, it is a good idea to structure the values ​​that the system must subsequently process.

Q3. What if my schema gets too big?

It's better to break it down into smaller pieces. Separate the extraction, classification, response, and review needs into step-by-step schemas rather than putting them into one large schema.

11. References and uncertainty

References

• OpenAI Structured Outputs:https://platform.openai.com/docs/guides/structured-outputs
• JSON Schema:https://json-schema.org/
• Pydantic:https://docs.pydantic.dev/

uncertainty

• Supported models and detailed API parameters may change over time.
• Performance and failure rates of complex schemas require real-world testing.

---