Building the Experience Loop

A defensive agent should retain what happened after the verifier returns a score. Each authorized attempt can become evidence, training data, or a new test.

Most pipelines keep the final score and discard the path behind it: wrong hypotheses, missing permissions, failed patches, and recovery steps. The next rollout then starts with almost none of that experience.

Tencent's recently published Hyra scaffold offers a useful pattern. A context agent maintains an Experience Bank, multiple proposal agents explore candidate solutions, fresh sandboxes execute them, and evaluators return scores and artifacts to the bank. The framework is deliberately simple: use search, execution, feedback, and memory rather than encoding every research strategy in the scaffold.

We can adapt this pattern to defensive cyber training. The key adaptation is to make authorization, containment, independent verification, and evaluator integrity part of the loop's definition.

The useful part is the loop around the model

Hyra wraps a model in a search loop with persistent memory. A context agent selects bounded prior evidence, proposal agents generate candidates, sandboxes execute them, and evaluators return scores and artifacts to the Experience Bank. The model can remain fixed while the surrounding system improves what it produces.

This is agentic search with persistent experimental memory. It produces material for later SFT or RL; PPO and GRPO remain separate policy-gradient training methods.

A defensive loop starts with authorization

A defensive cyber agent needs a broad authorized action space. Every task should carry a signed scope containing:

• target identities, environment IDs, permitted actions, and data boundaries;
• credential scope, tool permissions, and token, turn, cost, and time budgets; and
• approval requirements for sensitive transitions.

The context agent may decide which prior experiments are relevant. It may not expand the task's authority. Proposal agents inherit the same immutable scope, and the runtime enforces it without relying on the model's interpretation.

The loop has four operational parts:

1. Scope and context. Fix the objective, authorization, environment, budget, and bounded evidence before exploration.
2. Proposal and execution. Several policy copies explore candidates asynchronously in tiered sandboxes, from gVisor to isolated microVMs.
3. Independent verification. Check both the technical end state and the authorization path used to reach it.
4. Experience Bank. Store immutable evidence so the attempt can be reproduced and safely reused.

One attempt, three assets

A verified rollout can produce three different assets:

1. Grounded SFT examples

After cleaning and review, successful trajectories can become SFT data with their scope, observations, tool calls, recovery, verifier evidence, and provenance. Remove repetitive search, secrets, and unsupported reasoning. Failed attempts can become contrastive examples, error diagnoses, or stop-and-request-approval demonstrations.

2. RL rollouts

Once SFT is stable, the same environment can provide on-policy rollouts. Reward technical state, evidence, recovery, and resource use behind authorization and containment gates, so task success cannot compensate for a safety failure.

3. Regression evaluations

Unexpected failures should feed the private evaluation suite. Add canaries for hidden-patch copying, state-delta checks for disabled controls, and incident-derived tests for repeated credential searching. Training can learn from the pattern; frozen test instances and hidden answers remain excluded.

The verifier is the bottleneck

Search finds whatever the evaluator rewards, including shortcuts its designers did not anticipate. Tencent reports Hyra candidates leaking future information or caching outputs to obtain attractive scores without genuine progress.

In defensive tasks, the equivalent shortcuts include suppressing a vulnerable test, broadening an IAM policy, changing telemetry, or writing a persuasive report without changing system state.

The verifier should inspect independent state whenever the outcome is mechanically observable. An LLM judge can assess report clarity or the quality of an uncertainty statement. It should not replace a functional test, policy evaluation, packet trace, service-health check, or hidden security regression.

Evaluator evolution needs a protected boundary

Hyra's bilevel loop can improve both the solution and the evaluator. A cyber training system needs a protected boundary:

1. Development evaluator: fast, visible, and editable; used for dense feedback during exploration.
2. Frozen hidden verifier: immutable within a run; contains hidden tests, canaries, state checks, and authorization assertions.
3. Promotion process: human-reviewed and versioned; evaluator changes must pass historical incidents and untouched canaries before becoming authoritative.

The Experience Bank is more than an answer cache

Retrieving the nearest successful response encourages copying and contamination. Store the experimental structure instead: task and difficulty, authorization, model, policy, environment and verifier versions, actions, observations, artifacts, state deltas, scores, failure class, and reuse permissions. The context agent may retrieve abstractions and evidence, but held-out solutions, private flags, reference patches, and secret verifier behavior stay out of the bank.

Start with secure-code remediation

Start with a narrow secure-code pilot: executable tests, repository state, clear artifacts, and inexpensive resets.

• 300-500 owned or permissively licensed repository environments;
• one vulnerability and one explicit authorization object per task;
• four to eight proposal rollouts per task;
• fresh microVM state, functional and hidden security tests, and patch-minimality checks; and
• a frozen checkpoint with complete trace, cost, environment, and verifier versioning.

The first test is whether the loop beats a matched single-rollout baseline.

Measure pass@1 and success@k, verified gain per proposal, cost and wall-clock time, environment and verifier failures, reward-hacking discoveries, and held-out performance. Expand to detection engineering, cloud and identity policy, incident response, or multi-host ranges only after this loop is stable.

What could still go wrong

The architecture does not create trustworthy experience automatically. Shared models can produce correlated mistakes, the bank can amplify bad assumptions, context can leak held-out information, verifiers can overfit shortcuts, and flaky sandboxes can look like model failures. Version every component, preserve denominators, run multiple seeds, keep private temporal holdouts, and prove the harness before attaching an expensive RL trainer.

Accumulate evidence across attempts

A defensive agent becomes more useful when the system can convert authorized work into reproducible evidence, distinguish genuine progress from shortcuts, and reuse experience without contaminating its own evaluations.

The experience loop gives us a way to connect those pieces:

This gives us a narrower, more useful engineering objective than recursive self-improvement. Before asking an agent to improve itself, build a system that can tell when an improvement is real.

Further reading

• Hyra: A simple yet effective scaffold for general discovery
• The Bitter Lesson
• Evaluating Reliability Beyond Pass@k
• Training Security Models with Verifiable Outcomes