AI Memory Is Not RAG

Memory is one of the most easily exaggerated features in AI agents. Remembering all your logs might seem smart, but in reality, old assumptions and one-time failures can taint subsequent work.

Conversely, if nothing is remembered, the agent will repeat the same failure. So what is needed is not “many memories,” but “procedures to elevate them to verified memories.”

The conclusion of this article is simple.

Failures should first be left in the run ledger, and only repeatable and verified knowledge should be promoted to Memory.

Key takeaways

• RAG is a layer that searches for external knowledge, and Memory is a layer that passes verified repetitive knowledge to the next task.
• You must distinguish between Transcript, Failure Artifact, Memory Candidate, and Validated Memory.
• GitHub Copilot Memory emphasizes code location citation, current branch verification, and stale memory deletion policy.
• Claude Code separatesCLAUDE.mdand auto memory, and treats memory as context rather than enforced configuration.
• It is best to record failed output by task attempt rather than the entire goal.

Memory and RAG are different

RAG is a method of retrieving necessary knowledge and putting it into the current task.

For example, here is RAG:

Copy1API Documentation Search2Search previous PRs3Design Document Search4README Search5Issue Search

On the other hand, Memory is a layer that passes on project-specific repetitive knowledge learned while the agent is working to the next task.

The following are closer memory candidates.

Copy1This repository only uses pnpm.2You must run db:prepare before the integration test.3When modifying the billing module, the contract test must also be modified.4DB access that does not go through src/lib/db.ts is prohibited.

The two have different purposes.

Design principles demonstrated by GitHub Copilot Memory

[GitHub Copilot Memory document] (https://docs.github.com/en/copilot/concepts/agents/copilot-memory) explains that Copilot stores repository-specific memory, which can be utilized by the Copilot coding agent, code review, and CLI.

The important thing is that the memory is saved with a code location citation, verified against the current codebase and branch before use, and automatically deleted after 28 days to prevent stale memory.

There are three design principles to be gained from this:

Copy11. Memory must have a source.22. Must be verified against the current codebase.33. Old memory should expire.

This is what a good memory should look like.

Copy1Validated Memory:2Fact:3pnpm db:prepare is required before running the integration test.4 5Source:6- package.json scripts7- docs/testing.md8- CI workflow9 10Scope:11- test/integration/**12- Current repository13 14Validated At:152026-05-0616 17Expiration:18Re-verification after 30 days

Conversely, the following is not memory.

Copy1The test earlier failed.

This is a transcript or run ledger entry.

Separation of instructions and memory shown by Claude Code memory

[Claude Code memory document] (https://code.claude.com/docs/en/memory) explains memory files and auto memory, and explains that Claude treats them as context rather than enforced configuration. The/memorycommand provides a flow to check loaded memory files and auto memory settings.

This distinction is important.

Writing “Do not read secret” inCLAUDE.mdis a good instruction. But it's not a security policy. Security policies must be processed at the permission layer, which blocks actual file access, network access, production commands, secret output, etc.

Memory is context. Permission is enforcement. Mixing the two is dangerous.

Long-term memory is harder than you think

It may seem like creating a memory will solve all your problems, but in reality, it doesn't. Long-term memory is intertwined with temporal reasoning, causal reasoning, and multi-hop reasoning.

LoCoMo-related research evaluates long-term conversational memory, addressing temporal relationships, causal relationships, and multi-hop recall of long conversations. Coding agents have the same problem.

Old memory can actually be dangerous.

Copy1In the past, I used npm, but now it has been changed to pnpm.2The old test command is no longer valid.3The old DB path disappeared due to refactoring.

Therefore, the key to memory is not to store a lot, but to verify and store less.

Where should I leave the failed output?

In AI work, the unit of failure output is not the entire goal. There are multiple tasks within one goal, and failure occurs in smaller attempts.

Goal: Improve checkout flow

Task 1: Analyze existing payment flow

Task 1: Analyze existing payment flow

Task 2: Replace payment client

Task 2: Replace payment client

Task 3: Modify webhook handler

Task 3: Modify webhook handler

Task 4: Write tests

Task 4: Write tests

Task 5: Fix failing tests

Failure Artifact: task attempt unit

Failure usually occurs on a task attempt basis.

Therefore, the failure output should be left like this.

Copy1Bigger than Agent Turn,2Smaller than Goal,3Reviewable Task Attempt units

If you save it too small, there will be a lot of noise. If you save too much, it will be difficult to find the cause.

run ledger basic structure

The run ledger is a record of job execution.

Copy1run ledger Entry:2task_id: profile-form-validation3attempt_id: attempt-0034 5input:6- src/app/settings/profile/page.tsx7- src/features/profile/schema.ts8- pnpm test output9 10action:11- Changed client validation schema structure12- Modify form error mapping13- Run unit test14 15result: failed16evidence: pnpm test profile-form failed17diagnosis: client component references server-only module18next_action: Separate public validation schema into shared validation19memory_decision: pending. Currently, it is not promoted to memory as it is a one-time failure.

The key to run ledger is to separate “what was done” from “why it failed.” Especiallymemory_decisionis important. All failures should not be stored in memory.

Memory promotion policy

Memory promotion should be done conservatively. The entire conversation must go through a verification pipeline, not directly into memory.

1. 01Transcript
2. 02Failure Artifact
3. 03Memory Candidate
4. 04Validated Memory

Promotion criteria are as follows:

Copy1[ ] Has the same problem been repeated more than once?2[ ] Is it possible to check it in the current code base?3[ ] Isn't this a branch-specific temporary state?4[ ] Are there any source files or logs?5[ ] Is the scope of application clear?6[ ] Are there any expiration or revalidation conditions?

Development harness application example

Harness construction can start like this:

Copy1.agent-runtime/2goals/3profile-update/4goal.md5run-ledger.md6artifacts/7failure-attempt-001.md8test-result-002.md9memory/10candidates/11candidate-001.md12validated/13test-conventions.md

The important thing is not to make memory the agent's free notebook. Memory is the layer that affects the next execution. Therefore, it must be human readable and editable.

Copy1Must be able to read.2It should be possible to delete it.3The source must be identifiable.4It must be possible to expire it.

checklist

If the criteria below are not met before attaching the memory, it is safer to clean up the run ledger first.

Copy1[ ] RAG and Memory are distinguished.2[ ] Transcript and memory were distinguished.3[ ] Failure records are first left in the run ledger.4[ ] There are standards for Memory promotion.5[ ] Memory has a source.6[ ] Memory has a scope.7[ ] Memory has validated_at.8[ ] Memory has expiration.9[ ] Secret, personal information, and token are not stored.10[ ] People can audit and modify the memory.

Standards to be taken in this episode

Memory is neither a search index nor a full conversation log. Failures should first be left in the run ledger, and only facts that are repeatable and verified in the current code should be promoted to Memory. The standard for discarding memory is more important than the standard for storing it.

Next time

In Part 5, we organize the contents so far into a set of actual development harness documents. We provide templatesgoal.md,run-ledger.md,artifact-contract.md,memory-policy.md, andpermission-policy.mdthat you can copy and use right away.

Continue reading the series

• Part 1: Why are coding agents runtime?
• Part 2: Goal-based development through Codex/goal
• Part 3: Multi-agent development workflow from A2A and MCP perspective
• Part 4: AI Memory is not RAG
• Part 5: AI Coding Agent Document Set for Application to Development Harness

References

• About agentic memory for GitHub Copilot
• How Claude remembers your project
• Evaluating Very Long-Term Conversational Memory of LLM Agents

Comparison example: promote to memory vs. keep only in ledger

If you load all execution logs into memory, the prompt for the next task will become messy. Conversely, if you leave nothing behind, you will repeat the same failure. The criterion is reusability. A single test failure log is left in the run ledger, and work rules that have been repeated multiple times are raised as memory candidates.

Boundary cases are success experiences. The fact that "npm run testwas enough this time" shouldn't be immediately raised as a general rule. It is necessary to distinguish whether a rule is valid only when there is a content change in the same repo, or whether it is also valid when there is a routing change. Memory promotion is not a save button, but an operational decision that has been compressed and verified.