Connecting LLM Request Traces with OpenTelemetry

View API, RAG, model call, and response validation as one flow

In this article, we outline how to connect the request flow of the LLM service with a trace using OpenTelemetry.

Failure analysis for LLM services is more difficult than for regular APIs. This is because search, model call, response validation, storage, cache, and external provider call are mixed within one request. Users say “response is slow,” but the actual cause may be vector search, model call, or schema validation retry.

So we need a trace.

Analysis base date: 2026-05-12 Practice standard environment: FastAPI, OpenTelemetry, PostgreSQL, Redis, LLM Provider API Main reference materials: OpenTelemetry Docs, W3C Trace Context, Google SRE

Key takeaways

• OpenTelemetry is an observability framework for creating, collecting, and exporting traces, metrics, and logs.
• LLM requests must be divided into API, retrieval, LLM call, validation, and DB save into spans.
• trace_id must be connected to log, metric, and eval result.
• The original text of the prompt and the full text of the document are not included in the span attribute.
• You should look at p95/p99 latency, error rate, and token usage along with trace.

1. Why trace is needed for LLM services

The LLM request is divided into several subtasks.

Copy1# This is an example.2POST /answers3→ authenticate4→ rate limit check5→ retrieval6→ prompt build7→ llm call8→ schema validation9→ db save10→ response

If your overall response time is 8 seconds, you can't improve it if you don't know where the time was taken.

Trace shows this flow divided into spans.

2. OpenTelemetry basic concepts

In LLM services, it is important to connect trace and token usage metrics.

3. LLM request span design

Recommended span structure:

Copy1# This is an example.2answer.create3├── auth.check4├── rate_limit.check5├── retrieval.search6│   └── vector.query7├── prompt.build8├── llm.call9├── output.validate10└── answer.save

The duration and error status are recorded for each span.

4. trace ID propagation

If there is atraceparentheader coming from outside, it is inherited, and if not, a new trace is created.

Copy1# This is an example.2traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Including a request_id or trace_id in the response also makes tracking easier when receiving a crash report.

Copy1// This is an example JSON structure.2{3  "answer": "...",4  "trace_id": "0af7651916cd43dd8448eb211c80319c"5}

5. Span Attribute Design

A good attribute should be helpful for analysis and not contain sensitive information.

6. Sensitive information and security

Security Caution OpenTelemetry attributes and logs should not include the original text of the prompt, full text of the document, personal information, API key, or access token. Since observable data can also be transmitted to external systems, its security level must be managed separately.

Instead, record hash, length, and category.

Copy1# This is an example.2question_hash3prompt_version4context_token_count5retrieved_chunk_count6document_scope

7. Connect Metrics and Logs

It is difficult to see the overall trend using Trace alone. Metrics and logs must also be designed together.

Example metric:

Copy1# This is an example.2llm_request_duration_ms3llm_provider_latency_ms4llm_schema_validation_fail_total5rag_retrieval_latency_ms6rag_empty_result_total7llm_input_tokens_total8llm_output_tokens_total

Logs must include trace_id.

Copy1// This is an example JSON structure.2{3  "level": "ERROR",4  "message": "LLM schema validation failed",5  "trace_id": "0af...",6  "prompt_version": "answer.v3",7  "error_code": "SCHEMA_VALIDATION_FAILED"8}

8. What to see in Dashboard

Your initial dashboard doesn't have to be complicated.

Copy1# This is an example.2[ ] Number of requests3[ ] p50/p95/p99 latency4[ ] error rate5[ ] provider latency6[ ] schema validation failure rate7[ ] retrieval latency8[ ] token usage9[ ] cache hit rate10[ ] eval pass rate

Google SRE's four golden signals - latency, traffic, errors, and saturation - can be expanded to suit the LLM service.

9. FastAPI example structure

Copy1# This is example code.2from opentelemetry import trace3 4tracer = trace.get_tracer(__name__)5 6# This declaration shows an example flow.7async def create_answer(req):8    with tracer.start_as_current_span("answer.create") as span:9        span.set_attribute("llm.prompt_version", "answer.v3")10 11        with tracer.start_as_current_span("retrieval.search") as s:12            chunks = await retrieve(req.question)13            s.set_attribute("rag.top_k", len(chunks))14 15        with tracer.start_as_current_span("llm.call") as s:16            result = await call_llm(req, chunks)17            s.set_attribute("llm.input_tokens", result.usage.input_tokens)18            s.set_attribute("llm.output_tokens", result.usage.output_tokens)19 20        with tracer.start_as_current_span("output.validate"):21            validated = validate_output(result)22 23        return validated

This example is code to demonstrate the structure. In the actual service, middleware, exporter, and collector settings are required.

10. Practical checklist

Copy1# This is an example.2[ ] Can I view the entire request as one trace?3[ ] Are retrieval, llm call, and validation separate spans?4[ ] Is trace_id included in API responses and logs?5[ ] Is there no sensitive information in the span attribute?6[ ] Is token usage recorded as a metric?7[ ] Do you track schema validation failure?8[ ] Do you see p95/p99 latency in the dashboard?9[ ] Is it possible to connect eval result and prompt_version?

Failure case: There are a lot of logs, but the bottleneck cannot be found.

Even though the LLM service leaves many logs such asrequest started,retrieval done, andmodel done, it is sometimes difficult to analyze the failure. Although we know that each log belongs to the same request, it is difficult to see at a glance which step is pushing up the p95 latency, how many retries occurred, and whether the validation failure occurred before or after the model call. A log is a list of events, and a trace shows the parent-child relationship and time between events.

In particular, RAG requests are a bottleneck that changes every time. Some requests have slow vector search, some have long provider queues, and some require a second model call due to schema validation retry. If there are spansapi.request,retrieval.search,llm.call,output.validate, andanswer.persistin one trace, the problem location can be separated step by step.

Example implementation: LLM request span structure

Copy1trace llm_answer_request2span api.request3span auth.check4span retrieval.search5span llm.call6span output.validate7span answer.persist

Each span attribute contains an operable summary value rather than the original text.

Copy1llm.model = "runtime-selected-model"2llm.prompt_version = "rag_answer.v4"3llm.input_tokens = 42004llm.output_tokens = 6205retrieval.top_k = 86retrieval.index = "document_chunks_hnsw_v2"7validation.schema = "answer_with_citations.v2"

It is important not to include the full text of the prompt, the full text of the document, or user personal information in the attribute. Instead, it leaves the minimum values ​​necessary for reproduction and analysis, such as hash, token count, version, and chunk id.

Checklist application results

With this structure, a vague report of “LLM is slow” can be changed to “retrieval p95 has increased” or “schema validation retry has increased”.

Operational example: Narrowing down slow response reports to traces

Let's say a user reports that "it took 12 seconds to reply." If there is no trace, you will have to search through the logs chronologically. If you have a trace, you can immediately compare which spans spent time within one request.

In this trace, you can't just blame the provider. Both retrieval and validation are slow. Next checks are vector index status, metadata filter selectivity, and validation failure log. You should also check the metric to see ifrag_empty_result_totalorschema_validation_fail_totalhas increased during the same time period.

A borderline case is the masking of sensitive information. To quickly see the failure, put the prompt and chunk text into the span attribute, and later the observable storage becomes a private information storage. Instead, it is safer to leaveprompt_hash,chunk_ids,token_count, andschema_version, and handle reproductions that require the original text in a separate eval repository with permission. A good trace is less of an exhaustive log and more of a map that narrows down the cause.

11. Q&A

Q1. If I just leave a good log, is there no need to have a trace?

For small services, you can start with logs. However, if the request is broken down into multiple steps, trace is much more advantageous. In particular, a span is needed to distinguish between the LLM provider call and the retrieval bottleneck.

Q2. Wouldn't it be expensive to trace every request?

Sampling is available. However, it is recommended to preferentially preserve error requests, long-delay requests, and eval failure requests.

Q3. Can I put the contents of the prompt in the trace?

Not recommended. Prompts and document chunks may contain sensitive information. Replace with hash, token count, version, etc.

12. References and uncertainty

References

• OpenTelemetry Docs:https://opentelemetry.io/docs/
• What is OpenTelemetry:https://opentelemetry.io/docs/what-is-opentelemetry/
• W3C Trace Context:https://www.w3.org/TR/trace-context/
• Google SRE — Monitoring Distributed Systems:https://sre.google/sre-book/monitoring-distributed-systems/

uncertainty

• OpenTelemetry SDK settings vary depending on the language and framework.
• Sampling policies and data retention periods should be determined based on the organization's cost and security policies.

---