An Introduction to LLM Evals

Measuring quality that is not captured by test code

This article summarizes the basic concepts of LLM Evals and how to apply them in practice.

General backend testing is relatively clear. If the input is the same, the output should be the same, and the DB state should change as expected. But LLM services are different. Even the same question may be worded slightly differently, the answer may be plausible but wrong, and the quality of a particular case may decrease even if the prompt is only slightly changed.

That's why LLM services require a separate evaluation system.

Analysis date: 2026-05-12 Practice standard environment: OpenAI Evals concept, Python, pytest, JSONL dataset Main reference materials: OpenAI Evals Docs, OpenAI Evaluation Best Practices

Key takeaways

• Evals are structured tests that measure whether LLM output meets expected standards.
• Golden Set is a representative question dataset for regression testing.
• LLM quality evaluation should be divided into correctness, groundedness, citation accuracy, and safety.
• Prompt changes and model changes are dangerous if deployed without eval results.
• Evals are a complement to test code, not a replacement.

1. Why does LLM require a separate assessment?

Quality issues in LLM services are not well captured by general testing.

That is, the test code checks whether the system is broken, and eval checks whether the answer is usable.

2. Difference between Unit Test and Eval

You need both. Schema validation is considered unit/integration test, and answer quality is considered eval.

3. Creating a Golden Set

The Golden Set is a dataset that collects representative questions and expected standards.

Copy1// This is an example JSON structure.2{3  "id": "rag_001",4  "question": "When to use the Redis Cache Aside pattern?",5  "expected_facts": [6    "Check the cache first",7    "In case of a miss, the original data storage is searched.",8    "Store search results in cache with TTL"9  ],10  "must_cite": true,11  "category": "cache"12}

A good golden set should contain a variety of failure types.

Copy1# This is an example.2[ ] Easy questions3[ ] Ambiguous question4[ ] Questions not in the document5[ ] Questions that require up-to-dateness6[ ] Questions requiring permission7[ ] Citation is an important question8[ ] Questions that require a safety policy

4. Define evaluation criteria

Evaluation criteria should not be lumped together.

5. Grader design

Grader can be human, rule-based, or LLM-as-a-Judge.

It is safer to start with rule-based + human review initially and then use an LLM judge as an assistant.

6. Divide RAG evaluation

RAGs should be evaluated separately for retrieval and generation.

Copy1# This is an example.2Retrieval Eval:3Is the correct answer document in top-k?4 5Generation Eval:6Did you give the correct answer based on the retrieved document?

example:

7. Connect to Release Gate

Eval should be a standard for distribution, not just a report that you read and end.

Copy1# This is an example.2Prompt v3 deployment terms:3- overall pass rate >= 90%4- critical set pass rate >= 98%5- citation accuracy >= 95%6- regression count <= 3

Compare before and after changing prompts or models.

Copy1# This is an example.2baseline: prompt.v2 + model.A3candidate: prompt.v3 + model.A4candidate: prompt.v2 + model.B

8. Operational indicators and dashboard

This metric is used to determine deployment, improve quality, and record prompt changes.

9. Practical checklist

Copy1# This is an example.2[ ] Representative question Is there a golden set?3[ ] Are search evaluation and answer evaluation separate?4[ ] Did you distinguish between correctness and groundedness?5[ ] Do you evaluate citation accuracy?6[ ] Compare eval before and after changing the prompt?7[ ] Are critical cases managed separately?8[ ] Are eval failure cases reflected in the next improvement?9[ ] Is the eval result connected to the release gate?

10. Q&A

Q1. How often should I run Eval?

It is recommended to run this whenever the prompt, model, or retrieval logic changes. During operation, drift can be checked through periodic sampling.

Q2. Can I Trust LLM-as-a-Judge?

It's useful as an aid, but dangerous if you trust it completely. Critical cases should be combined with human review or rule-based checks.

Q3. How many Golden Sets should I start with?

At first, 30 to 50 is enough. The important thing is to include a variety of failure types rather than numbers.

11. References and uncertainty

References

• OpenAI Evals:https://platform.openai.com/docs/guides/evals
• OpenAI Evaluation Best Practices:https://platform.openai.com/docs/guides/evaluation-best-practices
• OpenAI Cookbook — Evals:https://developers.openai.com/cookbook/topic/evals

uncertainty

• Eval criteria must be tailored to the service domain.
• LLM-as-a-Judge results may vary depending on the judge model, rubric, and temperature.

---

Practical example: starting small with a golden set

There is no need to create hundreds of assessment questions from scratch. You might start with about 20 questions that come up frequently in operations, 10 questions that would be a big problem if they fail, and 10 questions that the new feature is intended for. The important thing is to write down the question, the basis for expectations, the range of acceptable responses, and the failure conditions.

A failure case is using only the LLM-as-a-Judge score as the release gate. The grader is also a model, so standards can be shaky. Important policy, legal, and payment responses should incorporate rule-based checks or human review samples. Eval is not a tool that automatically guarantees the correct answer, but rather a measurement device that allows you to compare changes in quality before and after deployment using the same criteria.

Even if it is a small golden set, a version must be added. It is impossible to interpret changes in scores without knowing whether the questions changed, the expectations changed, or the grader prompt changed. It is safer to set aside “questions that must never be failed” rather than average scores at the distribution gate.