Designing Production-Ready APIs

Failure responses and tracking IDs come before endpoints

This article summarizes how to design the LLM service's API in an operable form.

When you think of API design, URLs and HTTP methods usually come to mind. First decide on endpoints such asPOST /chatandGET /documents/{id}. However, from an operational perspective, there are things you need to decide before endpoints.

This is a failure response. Tracking ID. Whether retry is possible. Rate limit policy. It is based on health check and readiness.

Analysis date: 2026-05-12 Practice standard environment: FastAPI, Pydantic, PostgreSQL, OpenTelemetry Key reference materials: OpenAPI Specification, FastAPI Error Handling, W3C Trace Context, Google SRE Fact checking of official documents and operational interpretation of this article are separated in the main text.

Key takeaways

• API contracts include not only success responses but also failure responses.
• All requests must haverequest_idortrace_id.
• The LLM API must document retry, timeout, and idempotency.
• Health checks should separate process survival verification and dependency readiness.
• The API document is not just a front-end specification, but also a document for operators and failure responders.

1. What is an operational API?

An operational API is not an API that only works well under normal circumstances. This is an API that allows the cause of a failure to be tracked, the caller to determine whether to retry, and the operator to check the status with indicators.

In LLM services, this difference is more important. This is because model call failures, schema validation failures, rate limits, and long delays often occur.

2. Failure responses come before success responses.

Most APIs, which make it difficult to respond to failures in practice, have different failure responses.

Copy1// This is an example JSON structure.2{3  "message": "failed"4}

The cause cannot be determined from this response. I can't tell what the request is, if it's retryable, if it was sent incorrectly by the user, or if it's an internal server issue.

An operational failure response must include, at a minimum, the information below:

Copy1// This is an example JSON structure.2{3  "error": {4    "code": "LLM_SCHEMA_VALIDATION_FAILED",5    "message": "Model response did not match the expected schema.",6    "retryable": true,7    "details": {8      "schema_version": "answer.v1"9    }10  },11  "request_id": "req_01HX...",12  "trace_id": "0af7651916cd43dd8448eb211c80319c"13}

3. Error Response Envelope Design

The recommended error envelope structure is as follows.

One thing to be careful of is not to include the original text of the prompt, personal information, API key, or full text of internal documents indetails.

Security Caution LLM requests/responses may contain sensitive documents. You must first set a masking policy to avoid including the original text of prompts, document chunks, and user inputs in failure responses and logs.

4. trace ID and Request ID design

request_idis a single request identifier inside the service.trace_idis a distributed tracking identifier that cuts across multiple services and external calls.

Copy1# This is an example.2Client3→ API Server span4→ Retrieval span5→ LLM Provider span6→ Validation span7→ DB Save span

In the LLM service, a request is divided into several subtasks. So you need to connect a trace to see “where it’s slow.”

An example response header is as follows:

Copy1# This is an example.2X-Request-Id: req_01HXABC3traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

5. API that requires idempotency key

Not all APIs require an idempotency key. However, it is required for APIs that change state.

The Document Index API is a prime example. If the same document is indexed multiple times, vector rows may be duplicated and search quality may decrease.

Copy1# This is an example.2POST /documents/doc_123/index3Idempotency-Key: doc_123:index:v4

The server stores this key in the task table, and returns the existing task status when the same key comes again.

6. Rate limit response design

The LLM API has both cost and provider limitations. Therefore, the rate limit is not just a security device but also a cost control device.

Copy1# This is an example.2HTTP/1.1 429 Too Many Requests3X-RateLimit-Limit: 1004X-RateLimit-Remaining: 05X-RateLimit-Reset: <unix_timestamp>6Retry-After: 30

The 429 response must also include request_id and error code.

Copy1// This is an example JSON structure.2{3  "error": {4    "code": "RATE_LIMIT_EXCEEDED",5    "message": "Too many requests. Retry after 30 seconds.",6    "retryable": true7  },8  "request_id": "req_01HX..."9}

7. Health Check and Readiness

Having just one/healthis not enough.

You should be careful about whether to add the LLM provider status to readiness. If the entire service is excluded from traffic just because the external model API is temporarily slow, the failure may actually increase. Usually, the provider status is viewed as a separate metric and a fallback policy is set.

8. OpenAPI documentation standards

Your API documentation should include failure responses as well as success responses.

Copy1# This is an example.2responses:3'200':4description: Answer created5'400':6description: Invalid request7'429':8description: Rate limit exceeded9'500':10description: Internal server error

Items to document include:

Copy1# This is an example.2[ ] request schema3[ ] response schema4[ ] error schema5[ ] authentication6[ ] rate limit7[ ] retry policy8[ ] idempotency key9[ ] timeout expectation10[ ] trace/request ID header

9. Practical checklist

Copy1# This is an example.2[ ] Do all API responses have a request_id?3[ ] Is error.code a unique, searchable string?4[ ] Is there a retryable field?5[ ] 429 Is there a Retry-After in the response?6[ ] Is idempotency key supported in the state change API?7[ ] Have you separated liveness and readiness?8[ ] Does the OpenAPI document include a failure response?9[ ] Is there no sensitive information left in the log?

10. Q&A

Q1. Are both request_id and trace_id needed?

For small services, you can start with just request_id. However, if the service is divided into multiple components, trace_id is required. request_id is advantageous for tracing a single request, trace_id is advantageous for concatenating multiple spans.

Q2. Do I need to include an idempotency key in all APIs?

no. You can change the state and apply it first to the API with the possibility of retrying. APIs such as document indexing, payments, and external event handling are priorities.

Q3. Should LLM provider failure be considered a failure in readiness?

In most cases, it is better to view it as a separate provider health metric. Fallback, provider routing, and degraded responses may be safer than taking down the entire API server due to provider failure.

11. References and uncertainty

References

• OpenAPI Specification:https://swagger.io/specification/
• FastAPI Error Handling:https://fastapi.tiangolo.com/tutorial/handling-errors/
• W3C Trace Context:https://www.w3.org/TR/trace-context/
• Google SRE: Monitoring Distributed Systems:https://sre.google/sre-book/monitoring-distributed-systems/
• AWS Builders Library: Idempotent APIs:https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

uncertainty

• The actual error code scheme should align with your organization's API governance standards.
• The detailed implementation of the trace ID header may vary depending on the observability stack used.

---

Practical application example: API that sets the failure response first

If you are creating an LLM answer generation API, it is better to decide on failure responses before success responses. This is because whether a retry is possible depends on whether the user uploaded a document that is too large, a document without permission, a provider timeout, or schema validation failure.

Copy1{2  "error": {3    "code": "CONTEXT_TOO_LARGE",4    "message": "The input document has exceeded the current processing limit.",5    "retryable": false,6    "request_id": "req_123",7    "trace_id": "trace_456"8  }9}

The comparison standard isretryable. Rate limits can be retried after a certain period of time, but no permissions cannot be resolved by retrying. If the context is exceeded, the document must be shortened or the work divided. Without this distinction, the client would display all failures with the same alert, and the user wouldn't know what to do.