AWTL: Turning Failure Logs into Next-Run Hints

A good harness doesn't end with lots of failure logs. We need to turn the failure into a hint that can be used in the next run.

In Part 1, Ctx2Skill was applied from the operational memory perspective of the development harness, and in Part 2, it was summarized that MemoryGraph should only accept proven compact rules. The topic of this article is the middle layer.AWTL, or Agent Work Trace Logging.

The purpose of AWTL is not log collection. The goal is to attribute failures to the action unit and inject only as many recurrence prevention hints as necessary before the next attempt.

Key takeaways

• AWTL records agent actions in action/span/event units.
• Connect failed judge events with source actions, artifacts, and memory reads.
• The attribution results are compressed into failed turn cases.
• Before the next attempt, only relevant cases are injected into the Failure Prevention Brief.
• The replay scorecard filters out stale, risky, and blocked cases.
• MemoryGraph promotion is only allowed if it passes replay or human approval.

full flow

AWTL starts with a trace but does not end with a trace.

준비

1. 1Trace event collection
2. 2Failed judge event detection
3. 3Failure attribution
4. 4Failed turn case

반복 실행

1. 1Failure Prevention Brief
2. 2Replay scorecard

실패 시 다시 실행 단계로 돌아갑니다.

정리

1. 1Verified-only memory promotion

The point here is not “Paste all logs to the next prompt”. Pick only the failures that are relevant to the task at hand and turn them into small, actionable hints.

Minimum unit of trace event

It is difficult to know the cause of failure by only looking at the final PR. Failure usually begins already in the middle of the action.

Therefore, a trace must have at least the following layers:

Task

Run

Turn / Span

Action

Observation

judge_result

Trace Sink: Creating a Replayable Event Stream

Trace Sink is a layer that receives the work left by the agent during execution line by line and saves it as an execution log that can be read again later. The important thing here is not “save a lot,” but “save so that it can be analyzed again with the same criteria later.”

The trace referred to in this article is closer to a JSONL file. One event is included in one line.

Copy1{"type":"action","spanId":"s-42","tool":"shell","command":"npm test"}2{"type":"observation","spanId":"s-42","exitCode":1,"summary":"contract test failed"}3{"type":"judge_result","spanId":"s-42","status":"failed","reason":"contract mismatch"}

You must have this file so that you can later recalculate “what observation came after what action and why the judge considered it a failure.” So Trace Sink's first responsibility is twofold.

The code below is a boundary that prevents trace from being written to any directory.EXPECTED_TRACE_ROOTis the storage location allowed by the harness, and if any other path comes in, it will fail immediately.

Copy1function assertExpectedTraceRoot(traceRoot) {2  const resolvedRoot = normalizeAbsolutePath(traceRoot);3  const expectedRoot = normalizeAbsolutePath(EXPECTED_TRACE_ROOT);4 5  if (resolvedRoot !== expectedRoot) {6    throw new Error(`Invalid trace root: ${traceRoot}`);7  }8 9  return resolvedRoot;10}

The second problem is a broken JSONL line. If the process dies mid-execution or file writing is interrupted, a single line may not be parsed as JSON. At this time, if you discard the entire trace, even normal events will be lost. Therefore, only broken lines are separated intoquarantinefiles, and normal lines are continued to be analyzed.

Copy1quarantined.push(quarantineLine(2  quarantinePath,3  rawLine,4  reason,5  canonicalPath,6  index + 1,7));

This design ensures three things:

Harness Capture: Connecting action and judge

If the judge failure remains simplyfailed, it is difficult to use to prevent recurrence. You need to connect which action created which artifact and which verifier saw that artifact and failed.

Copy1async function recordJudgeResult(details = {}) {2  return emit("judge_result", {3    judge_name: toText(details.judgeName, "phase-verifier"),4    result: toText(details.result, "warn"),5    lifecycle_event: "judge_result",6    source_action_id: actionId,7    artifact_refs: normalizeArtifactRefs(details.artifactRefs ?? [], repoRoot),8    detail: toText(details.detail, ""),9  });10}

source_action_idandartifact_refsare the key. Without these two, failures are difficult to reproduce or prevent in subsequent runs.

Failure Attribution: Separating cause from promotability

Attribution connects failed judge events with source actions, artifacts, and memory reads. At the same time, we need to isolate whether this failure is subject to memory promotion.

Copy1return {2  failureEvent,3  failureTurnId,4  failedArtifactRefs,5  sourceActionIds,6  verifierActionId,7  touchedActionIds,8  memoryReadNodeIds,9  evidenceRefs,10  rootCauseSummary,11  verificationProbeCandidate,12  classification,13  failureTypeInfo,14  attributionHeuristics,15};

What is important is not the single root cause statement. This isclassification.

Long-term memory is faulty if we do not distinguish between environmental and agent failures. For example, if e2e fails because the browser binary is missing, you should not have memory saying “e2e is omitted in this repository.”

Failed Turn Case: Compact case for next run

Attribution results should be compressed into compact cases rather than the entire raw trace.

Copy1{2  "schema_version": 1,3  "case_id": "case-demo-a",4  "turn_id": "turn-3-1",5  "failure_turn_id": "turn-3-1",6  "failure_event_id": "judge-17",7  "artifact_refs": ["artifact:build-output"],8  "memory_read_node_ids": ["memory:contract-rule"],9  "prevention_hint": "Before closeout, run the same verifier again for the changed artifact.",10  "applicability": ["contract_change", "public_api"],11  "evidence_refs": ["judge:contract-verifier"]12}

Verification must also be rigorous.

Copy1if (caseValue.turn_id !== caseValue.failure_turn_id) {2  errors.push("turn_id and failure_turn_id must match");3}4 5if (!Array.isArray(caseValue.artifact_refs) || caseValue.artifact_refs.length === 0) {6  errors.push("artifact_refs must be a non-empty array of strings");7}

The case affects the next attempt. Therefore, cases withoutartifact_refs,evidence_refs, andapplicabilityare difficult to use as recurrence prevention data.

Failure Prevention Brief: Inject Smallly

You should not add all failed cases before the next attempt. Only cases that match the current phase should be included.

Copy1const selectedCases = selectFailurePreventionCases(loaded.cases, context, options);2 3if (selectedCases.length === 0) {4  return {5    status: "no-op",6    section: "",7  };8}

The brief should be short.

Copy1Failure Prevention Brief2- [high-confidence] Re-run the verifier that failed before Closeout as the target for the changed artifact.3- [scope: public_api] Update contract artifact and verifier evidence together.

A good brief satisfies three conditions:

Replay Scorecard: Managing the effectiveness of failure memories.

Even a failed case that was once valid can become stale over time. The verifier may change, the code structure may change, or the failure may no longer be reproducible.

Copy1{2  "schema_version": 1,3  "record_id": "replay-demo-a",4  "status": "passed",5  "decision": "allow_brief_and_promotion",6  "candidate_id": "memcand-demo-a",7  "case_id": "case-demo-a",8  "validated_by": "replay",9  "last_validated_at": "example-timestamp",10  "memory_graph_status": "candidate",11  "replay_status": "passed",12  "risk_level": "low",13  "applies_to": ["public_api"],14  "does_not_apply_to": ["internal_refactor"],15  "evidence_refs": ["judge:contract-verifier"]16}

Brief Check exclusion conditions with scorecard before injection.

Copy1export function isReplayScorecardExcluded(record = {}) {2  const status = normalizeStatus(record.status ?? record.result ?? record.outcome);3 4  return isReplayScorecardStaleOrRisky(record)5    || ["blocked", "skipped", "unavailable", "denied"].includes(status);6}

This filter prevents old failure memories from persisting in the prompt.

Memory promotion is the final step

The results from AWTL should not go directly into MemoryGraph. Promotion is the final step.

Copy1if (!approval.approved && !replayOk) {2  reasons.push("replay or human approval is required before promotion");3}4 5if (isImportedOnlyCandidate(candidate)) {6  reasons.push("imported-only or trace-only candidate is blocked");7}8 9const shouldWrite = options.writeMemoryGraph === true10  && toText(options.autoPromote, "verified-only") === "verified-only";

The policy created by these three conditions is clear.

• No promotion without replay or human approval.
• Imported-only candidates are blocked.
• Do not write to MemoryGraph without an explicit write flag.
• Auto-promotion only allows verified-only.

Practical Checklist

Before attaching the AWTL to the actual harness, check the following items:

Copy1AWTL application checklist2 3[ ] The event schema includes action, observation, judge_result, and artifact_ref.4[ ] Trace event has run_id, attempt_id, and turn/span/action identifiers.5[ ] Guarantees sortability with writer_seq or ingest_seq.6[ ] The trace root is safely fixed and path traversal is blocked.7[ ] Quarantine corrupt JSONL lines without ruining the entire trace.8[ ] judge_result has source_action_id and artifact_refs.9[ ] Include memory_read event in attribution.10[ ] failure attribution classifies the failure class.11[ ] environment/flaky/harness failure blocks default promotion.12[ ] The failed turn case only has compact metadata.13[ ] The prevention brief injects only cases that match the current phase.14[ ] Exclude stale/risky/blocked cases from the replay scorecard.15[ ] MemoryGraph promotion is limited to verified-only.

finish

The point of AWTL is not to keep good logs. It's about turning failure into a hint for the next move.

1. 01failure log
2. 02failure attribution
3. 03failed turn case
4. 04prevention brief
5. 05replay scorecard
6. 06verified-only memory promotion

Once this loop is created, the harness is not a simple executor. It becomes an operating system that observes execution, interprets failures, and promotes only verified knowledge to long-term memory.