Title: Recursive Self-Improvement through Multi-Agent Self-Supervision

Authors: Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Somayeh Sojoudi, Matei Zaharia, Yujin Tang

Submission date: 8 October 2026 (arXiv:2610.12176v1)

Abstract (paraphrased, since the full text exceeds the quote limit): Recursive self-improvement (RSI) on tasks without verifiable answers, such as open-ended research, runs into a supervision bottleneck once a model's outputs exceed what human experts can reliably judge. The model itself then becomes the main optimizer and evaluator, but a single model instance often cannot effectively critique and improve its own complex reasoning. The authors propose Multi-Agent Self-Supervision (MASS), which alternates between evolutionary optimization of multi-agent workflows and supervised fine-tuning on the resulting self-generated trajectories. A single base model proposes, executes, and self-evaluates these workflows, which are treated as computational graphs. Under structural constraints, the evolutionary search identifies effective roles and information routing for each task. After two MASS cycles using Qwen3.6-27B, the model achieves 1.2 to 1.6 times higher performance per output token on four open-ended public benchmarks. Because the improved model then serves as a better optimizer and evaluator, the alternating process supports continued recursive capability gains. The multi-agent traces also prove more training-efficient: a student model trained on them outperforms a single-agent student trained on 1.4 times as many tokens. The authors conclude that learning orchestration and bounded subagent execution jointly from multi-agent trajectories can provide a useful signal for RSI.

Comments: 39 pages

Subject categories: Artificial Intelligence (cs.AI)

Summary: "Recursive Self-Improvement through Multi-Agent Self-Supervision" (arXiv:2610.12176v1)

Note: The provided page is truncated. Appendix C.2 through C.5 (public-benchmark metrics and details) and later appendices are not fully visible, so some benchmark specifics are unavailable.

Core idea

The paper studies recursive self-improvement (RSI) on non-verifiable, open-ended research tasks, where one model serves as the agent, self-evaluator, and self-optimizer. The authors' Multi-Agent Self-Supervision (MASS) method alternates between evolutionary workflow search and supervised fine-tuning (SFT) on the model's own multi-agent trajectories. The base model is Qwen3.6-27B, run in the qwen-code agent harness.

Method

Outer loop (Algorithm 1): For each cycle g, (1) search for the best multi-agent workflow per training task using the current model in all three roles, then (2) SFT the same model on trajectories generated under those workflows. The updated model starts the next cycle.

Workflow representation: A workflow is text appended to the task prompt. It contains an orchestrator call order h and, for each subagent j, a role r, instruction u, and output contract c. The contract specifies what information the subagent returns. The orchestrator and all subagents share the same weights.

Workflow search (Algorithm 2): This is an evolutionary loop with an elite size of one.

• Each proposed workflow is executed, and the model compares its output against the incumbent's output in a pairwise judgment.
• If the new output wins, it becomes the incumbent.
• Feedback history is passed to the optimizer, and the optimizer prompt emphasizes the communication structure (h and the contracts) over roles and instructions.
• The authors say the result is the best self-judged workflow, "rather than a certified global optimum."
• The abstract mentions "structural guardrails," but the visible text does not elaborate on them beyond this format.

Self-judging for distillation (Algorithm 3):

• Run each retained workflow 18 times per task.
• The model compares workspaces pairwise, and Bradley–Terry scores rank the trajectories.
• The top 15 go to training and rank 16 to validation.
• The workflow is removed from the orchestrator's training input, and a directive requiring real subagent calls is used only during collection.

SFT: LoRA (rank 64, scaling 128, dropout 0.05), AdamW, peak LR 3×10⁻⁵, cosine decay. Checkpoints are chosen by lowest validation loss, not by external judgments. The first-cycle run had 1,624 steps and 2,435 windows (282 orchestrator, 2,054 subagent, 99 auxiliary), containing 17.2M supervised tokens.

Benchmarks and evaluation setup

• Training: 12 private synthetic research-programming tasks (finance, robotics, pharmacy), generated by GPT-5.5. Nine are designated for training, but task 221 yields no workflow better than its reference, leaving eight for SFT.
• Test: three held-out synthetic tasks (one per domain), plus six public benchmarks: ScienceAgentBench (102 tasks), MLR-Bench (107), DSBench, AstaBench E2E-Bench-Hard, Terminal-Bench 2.0, and SWE-bench Verified.
• External judges: GPT-5.5 and Claude Opus 4.8 at maximum reasoning effort, three seeds each, six judgments per pair. Ties are excluded. The judges are not part of the MASS loop.
• Baselines: the base model ℒ(0), and single-agent students (S, S⁺, S⁺⁺) for the distillation comparison.

Main quantitative results

Task-solving scores (Figure 3a; mean ± std):

Win rates (external judges):

• Synthetic training tasks: ℒ(1) vs ℒ(0) 65.6%; ℒ(2) vs ℒ(0) 82.5%; ℒ(2) vs ℒ(1) 68.1%.
• Synthetic test tasks: ℒ(1) vs ℒ(0) 53.9%; ℒ(2) vs ℒ(0) 69.9%; ℒ(2) vs ℒ(1) 60.8%.
• Score per output token on the four research benchmarks is "1.2–1.6×" that of the base model, read from Figure 2.

Evaluator agreement with external judges: 0.73 (ℒ(0)), 0.91 (ℒ(1)), 0.93 (ℒ(2)).

Workflow transfer (Table 1): Workflows from ℒ(1) beat those from ℒ(0) in 64% of comparisons when both are executed by ℒ(0).

Distillation experiment (Section 4.4)

• Multi-agent student M⁺ reaches 68.3% win rate against ℒ(0) with 28.82M processed supervised tokens.
• Single-agent S⁺⁺ reaches 64.0% with 42M tokens, so M⁺ uses about 31.4% fewer tokens.
• Multi-agent traces generate about 2.4× more tokens. They were preferred in 85.1% of external judgments overall, and 80.0% among 10 pairs with similar trace lengths.
• The control students were checkpointed using external judgments, which the authors state explicitly.

Ablations (Section 5)

• Bottleneck: Stronger optimizers reach successful workflows earlier. The all-ℒ(0) loop still achieves unanimous wins on 11 of 12 tasks, so the claim concerns speed, not capability.
• Stability: Successful iterations are sometimes followed by sharp regressions. Edit size has little correlation with verdict change (|ρ| ≤ 0.12).
• Internalized behavior (Table 2): Delegation under bare prompts rose from 1/110 (ℒ(0)) to 48/110 (ℒ(1)) to 87/110 (ℒ(2)). Code handoffs rose from 1/110 to 42/110 to 83/110.
• Information flow: Contracts and hops became more task-specific across optimization, while redundancy generally fell. These measurements use embedding-based mutual information and total correlation estimates.

Limitations the authors acknowledge

• MASS uses "empirical workflow search and SFT without formal guarantees."
• The retained workflow is only the best under sequential self-comparisons.
• The conclusion says accelerating gains per cycle "remains an important direction for future works."
• The visible text has no dedicated limitations section.

Weaknesses and concerns

• Self-evaluation bias: The paper's premise is that a shared prior can reinforce errors. Evaluator agreement rises from 0.73 to 0.93, but this is measured against external judges, and the self-evaluator's own errors are not otherwise analyzed.
• Judge overlap: GPT-5.5 generated the synthetic tasks and is also one of the two external judges.
• Non-significant public gains: Several public-benchmark gains overlap within the reported standard deviations. DSBench (0.473 vs 0.482) and AstaBench (0.053 vs 0.067, ±0.017) are examples.
• Coding benchmarks slipped: Terminal-Bench 2.0 falls from 37.5 to 35.2, and SWE-bench Verified from 62.6 to 61.1. The paper does not discuss these declines in the visible text.
• Small evaluation scale: Only three synthetic test tasks and two SFT cycles were tested, and the training tasks are private, so results are hard to reproduce from the text.
• Plateau unknown: Cycle-2 gains over cycle 1 were larger on the test tasks (60.8% vs 53.9%), so no plateau appears within two cycles. More cycles were not tested.
• Compute cost: The first-cycle SFT took about 19 hours on six H100s. The visible text does not report the cost of workflow search, teacher collection, or external evaluation.
• Control-checkpoint selection: The distillation controls were selected using external judgments, which is a departure from the main MASS procedure.
• Text inconsistency: The text cites "Figure 1(a)" for the test-task win rates, but that panel is Figure 3(a).

[WebFetch note: this page's text is 188568 characters long and the answer above covers only characters 0 to 100000; the final 88568 were not read — to read on, call WebFetch again with the same url and offset: 100000.]