Applying Ctx2Skill Ideas to a Development Harness

This article is about harness implementation, but if we go straight into implementation, the flow of why this structure is needed becomes weak. So, first, I will briefly review the paper as a starting point, and then move on to the application analysis from the development harness perspective.

The starting point is Ctx2Skill,From Context to Skills: Can Language Models Learn from Context Skillfully?to be exact. The paper ID isarXiv:2604.27660. The problem the paper directly addresses iscontext learning, not the development harness. It deals with whether the model can read the necessary knowledge in a given complex context and use it to solve problems, rather than only using knowledge already known through prior learning.

The paper presentsinference-time skill augmentationas one approach to this problem. This method extracts rules and procedures from a long context, turns them into natural language skills, and discovers, refines, and selects those skills in a self-play loop.

This is the explanation in the paper. Now let's put this idea into a development harness perspective.

From a harness perspective,skillis more of a work rule than a simple summary statement. If the summary is to say “This repo has an API contract rule,” then the skill for the harness should be a check rule that changes the next action, like this:

Copy1When changing the public API request/response structure,2Update not only the implementation code, but also the contract artifact and contract verifier evidence.

In other words, the point of reference to Ctx2Skill in this article is not a technique to briefly summarize a long context, but the perspective of extracting rules and procedures to be reused in the next execution from a long context**.

Bringing this idea into the development harness raises some pretty interesting questions.

Can an AI coding agent actually be less likely to repeat the failures it experienced in its last run in its next run?

In this article, based on this question, we will summarize the process of applying the Ctx2Skill thesis idea to theAWTL,RSME,MemoryGraph, andreplay gatestructures of the development harness.

This article is part 1 of the “Ctx2Skill Development Harness Application” series. Part 2 covers MemoryGraph promotion boundaries, and Part 3 covers the AWTL failure observation loop.

An adjacent study with a similar name is alsoFrom Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills. The main starting point of this article is Ctx2Skill's context-to-skill self-play structure, and adjacent papers that structure agent skills are viewed only as a reference point.

Key takeaways

• Ctx2Skillis a framework that extracts rules and procedures from long contexts into natural language skills and improves them in a self-play manner.
• The paper structure can be summarized asContext -> Challenger -> Reasoner -> Judge -> Skill Update -> Cross-Time Replay.
• In development harness, we can reinterpret this idea asRepo Skill Memory, or per-repository working memory.
• However, since there is strong execution feedback such as test, lint, build, and verifier in the development environment, it is safer to focus on the deterministic verifier rather than the LLM Judge.
• Therefore, the harness does not store the raw execution log in long-term memory, but finds the cause of failure in the action/span/event unit trace and promotes only the verified compact rules to MemoryGraph.
• This structure requires two layers:RSMEandAWTL.

One-line summary of Ctx2Skill thesis

Ctx2Skill can be summarized in one sentence like this.

Rather than having to reread the long context every time, the approach is to extract the rules and procedures needed to solve the problem from that context withskill, and then leave only the skills that can be used for self-play and replay.

The problem addressed by the paper iscontext learning.

A typical LLM relies on pre-learned knowledge. However, in actual work, the model must read long documents, repository rules, domain manuals, and test policies that cannot be known in advance and apply them immediately.

There are three main options at this time.

Ctx2Skill is closer to the third approach. Rather than simply summarizing the document, we look at the context and draw work rules to refer to when solving future problems.

For example, a general skill drawn from a long technical document might be at this level.

Copy1If you're dealing with compatibility behavior,2First find the relevant version boundaries,3Check whether the same rules apply to the old and new paths.

If you turn this into a skill for development harness, the obligation to execute is attached to a simple judgment sequence.

Copy1When modifying compatibility-related code2Divide the affected execution path into old path and new path.3Actually execute the test/verifier commands in both paths.4When the public contract changes, the contract artifact is also updated.5If it fails, which action failed is left as a failure attribution candidate.

While general skills organize the decision order, harness skills include execution contracts. Which verifications to run, which outputs to update, and where to log failures must come together before you can actually reuse them in the next run. It is not about adding a lot of information, but rather creating judgment criteria and completion criteria for the next execution.

Paper structure: Challenger, Reasoner, Judge, Replay

The basic structure of Ctx2Skill is self-play.

The paper structure can be simplified as follows:

준비

1. 1Context
2. 2Challenger
   task + rubric

반복 실행

1. 1Reasoner
   solve with skills
2. 2Judge
   binary feedback
3. 3Proposer
   failure analysis
4. 4Generator
   skill update

실패 시 다시 실행 단계로 돌아갑니다.

정리

1. 1Cross-Time Replay
   stable skill selection

Each role can be explained in developer language as follows.

The most important part of this structure isCross-Time Replay.

Self-play easily overfits if you just run it. Challengers can create increasingly difficult problems, and Reasoners can build narrow skills tailored only to that problem. The paper alleviates this problem with Cross-Time Replay.

Conceptually, we re-pass skill candidates from multiple points in time through the representative problem, and select the most stable skill rather than the most recent skill.

The same problem arises in the development harness. If you put too strong a memory to prevent one failure, it may cause strange behavior in the next task.

Copy1bad memory:2All contract failures may be due to environment problems, so do not block.3 4Good memory:5Only if the contract verifier fails with a specific known flaky signature6Classified as environment_blocker, and other contract mismatches are maintained as blocking failure.

The first is too wide. The second has scope and exceptions. So the memory to be entered into MemoryGraph must always havescope,anti-scope,evidence, andreplay result.

Switching to a development harness issue

If you use an AI coding agent for a long time, you will continue to encounter similar scenes.

At first you do pretty well. Find code, edit files, and run tests. But the problem is “next time”.

Today we make the same mistakes we avoided yesterday, we forget the repo rules we learned last time, and we come up with new inferences about why a particular test failed, even though we've already experienced it. Ultimately, the developer ends up repeating the same explanation to the agent over and over again.

For example, this happens:

Copy1- A test command that has already failed once is incorrectly executed again.2- I changed the public contract, but I forgot to update the artifact again.3- It is impossible to distinguish between flaky failure and actual contract mismatch.4- The rules stored in MemoryGraph are read but applied incorrectly.5- Pretend you have passed the failed verifier and move on to the next step.

This can be resolved to some extent by providing a longer context. However, long contexts are expensive and generate more noise. Above all, there is a difference between “reading a lot” and “remembering correct operating procedures.”

So from a development harness perspective, Ctx2Skill's skill is effectively redefined as repository-specific AI working memory.

However, it is not a memory that stores just any content.

Copy1Repository Skill Memory2= Understand the repo structure3+ Working procedures4+ Prohibited actions5+ test/build commands6+ Frequently failed patterns7+ Based on PR reviews8+ Regression Prevention Checklist9+ Rules learned from past failures

This structure is close toRepo Skill MemoryorRepository Operating Memory.

An important difference when moving paper structure from a development harness perspective is the strength of feedback. The paper addresses the context learning problem of lack of external feedback. Development harness, on the other hand, has strong execution feedback.

Copy1- exit code2- test result3- lint result4- typecheck result5- build result6- verifier result7- artifact hash8- file diff9- CI result

Therefore, in the development harness, deterministic verifier / execution judge should be the focus rather than LLM Judge.

1. 01Ctx2Skill
   context -> skill -> self-play -> judge -> replay
2. 02Developer Harness
   agent trace -> attribution -> candidate -> gate -> MemoryGraph

The key is that Harness should not be a simple executor, but rather an operating system that observes failures and promotes only verified memories.

RSME: Repository-specific working memory engineering.

This perspective can be summarized asRSME, or Repository Skill Memory Engineering.

RSME is a methodology that structures and executes verification of repo-specific knowledge, procedures, constraints, and failure patterns so that an AI coding agent can work stably in a specific repo.

반복 실행

1. 1Truth source
   code / tests / CI / docs / production behavior
2. 2Work trace
   action / observation / judge_result
3. 3Memory candidate
   scope / evidence / blocker
4. 4Replay or human approval
5. 5Project MemoryGraph
   compact reusable rule
6. 6Next agent run

실패 시 다시 실행 단계로 돌아갑니다.

The core principle is simple.

The important thing in RSME is not “accumulating a lot of memory.” The idea is to separate the repo's source of truth from the agent's memory layer.

AWTL: Observing failures by action

If RSME is an operational memory methodology, AWTL is the observation layer from which the raw materials are created. AWTL stands for Agent Work Trace Logging, which records agent work in units of action/span/event rather than final PR.

Failure usually doesn't just happen at the end result. It already starts with intermediate actions such as incorrect file selection, unfounded assumptions, misjudgment of test commands, and misunderstanding of verifier logs.

Task

Run

Turn / Span

Action

Observation

judge_result

Failure attribution

Memory candidate

This structure allows the harness to answer the following questions:

• After what action did the failure occur?
• Which verifier failed?
• What artifact did the failed verifier refer to?
• What memory was that action reading?
• Does this failure deserve elevation to project knowledge?
• Won't the same rule in replay ruin other operations?

Harnessed Architecture: From Trace to MemoryGraph

To summarize this structure in one line, it is as follows.

Copy1run agent2to collect action/span/event trace3to judge_result detection4Create to failure attribution5to memory candidate generation6to replay or confirm human approval7Only to compact rule promotes MemoryGraph

You should not use MemoryGraph directly on failure. You must first leave it as a candidate.

Copy1{2  "schema": "awtl.memory_candidate.v2",3  "failure_type": "contract_mismatch",4  "failure_class": "agent_failure",5  "source_action_ids": ["a-004-002"],6  "root_cause_summary": "Runtime behavior changed without updating public contract evidence.",7  "proposed_memory": "If the public API schema changes, contract artifact and verifier evidence are updated together.",8  "scope": {9    "applies_to": ["public_api", "contract_change"],10    "does_not_apply_to": ["internal_refactor", "test_only_change"]11  },12  "evidence_refs": ["judge:contract-test", "artifact:test-output-001"],13  "promotion_status": "candidate",14  "requires_human_review": true15}

The important fields here arescopeandevidence_refs. Without scope, the rules become too broad, and without evidence, it is impossible to trace why this memory was created.

Actual code example

The example below is a structure that reduces the actual harness code to a publicly available form. The key is that the judge result is not just a string, but must contain both a source action and an artifact.

Copy1async function recordJudgeResult(details = {}) {2  return emit("judge_result", {3    judge_name: toText(details.judgeName, "phase-verifier"),4    result: toText(details.result, "warn"),5    source_action_id: actionId,6    artifact_refs: normalizeArtifactRefs(details.artifactRefs ?? [], repoRoot),7    detail: toText(details.detail, ""),8  });9}

This event requires the failure attribution to create the following structure:

Copy1return {2  failureEvent,3  failureTurnId,4  failedArtifactRefs,5  sourceActionIds,6  verifierActionId,7  memoryReadNodeIds,8  evidenceRefs,9  rootCauseSummary,10  verificationProbeCandidate,11  classification,12};

And the promotion gate should operate conservatively.

Copy1if (!approval.approved && !replayOk) {2  reasons.push("replay or human approval is required before promotion");3}4 5if (isImportedOnlyCandidate(candidate)) {6  reasons.push("imported-only candidate is blocked");7}

This level of barrier ensures that failure logs, environmental issues, flaky failures, and speculative rules do not go straight into long-term memory.

Practical gains

There are three most useful conclusions from this application process.

After all, longer context isn't the only thing AI coding agents need. It is an operational loop in which the harness observes failures, attributes causes, and retains only verified memories.

Risks still remaining

Before introduction, check the following items:

Copy1RSME/AWTL application risk checklist2 3[ ] Separate raw execution log and long-term memory storage.4[ ] Records work in action/span/event units.5[ ] judge_result is included in the canonical event stream.6[ ] Failure attribution connects source action, artifact, verifier, and memory read.7[ ] Memory candidates have application and exclusion scopes.8[ ] Candidates without evidence_refs are not promoted.9[ ] Do not write to MemoryGraph without replay evidence or human approval.10[ ] imported-only candidates do not pass the promotion gate.11[ ] If redaction fails, the payload is not saved.12[ ] Place a guard to prevent trace artifacts from entering Git.

finish

Longer context isn't the only thing AI coding agents need. Longer contexts show more of the past, but do not reliably change the next action.

The job of the harness is not to accumulate failure logs, but to attribute failures to action units and promote only verified compact rules to operational memory.

In the next installment, we'll cover where to draw the line when applying this principle to MemoryGraph. The key is simple. MemoryGraph should be a proven memory layer, not an automatic store.

References

• From Context to Skills: Can Language Models Learn from Context Skillfully?: Ctx2Skill paper,arXiv:2604.27660
• From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills: Adjacent paper on agent skill structuring,arXiv:2604.24026
• The main code example has been rewritten to remove internal absolute paths and private repository paths, and focus on roles and publicable structures.