Reviewed L5 meta-improvement branch
Public
Reviewed L5 meta-improvement branch
Agent
Codex
Model
GPT 6 Astra
Started
Sep 14, 11:50 PM
Messages68
Downloads0
Workspace
Reviewed full branch diff against main.
UnderstandChecked documented invariants against scripts.
VerifyTested evaluation gate behavior with policy metadata changes.
ReproduceIsolated committed HEAD for consistent verification.
UnderstandYou are an independent reviewer with no stake in prior conclusions. Review the ENTIRE branch feat/l5-meta-improvement against origin/main (run git diff origin/main...HEAD — do NOT limit yourself to the latest commit). This branch adds an "L5" meta-improvement layer: a versioned improvement policy (docs/improvement-policy.json), scripts under scripts/ (improvement_policy.py, detect-recurring-pattern.py, archive-round.py, measure-policy-validity.py, revise-improvement-policy.py, mine-trace-failures.py, render-rsi-dashboard.py and their _test.py), workflows under .github/workflows/ (codex-review.yml, revise-improvement-policy.yml, archive-and-recommend.yml), and docs (docs/plans/recursive-meta-improvement.md, docs/rsi/).
Threat model: the meta-improver (an AI) may only rewrite the policy it owns; it must never be able to widen its own authority, game its own evaluation (coverage/validity), corrupt the archive or verifier, or accept a revision on evidence that does not support it. Findings should be things that are actually wrong, with a concrete reproduction. Run commands to verify: python3 -m pytest scripts/ -q -p no:cacheprovider, ruff check scripts/, and targeted python3 - <<'PY' ... PY reproductions. Never claim a result you did not observe.
Output format: a numbered list of findings, each tagged [P1] (must fix), [P2] (should fix) or [P3] (nit), each with file:line, the failure scenario, and how you reproduced it; then a "## Verification" section listing the commands you ran and their results; then a "## Verdict" line: whether the branch, as a whole, holds its stated invariants. If you find nothing in an area, say so explicitly.
SCOPE A — the Python scripts and their tests (scripts/*.py). Check especially: policy hash/versioning, rollback and ancestor logic, the evidence-binding rules (definitions, name@tag variants, truncation, undated traces), the wait gate on unjudged revisions, topic mining bounds, output guards (assert_ai_may_write roles, assert_safe_output), the failure miner's false-positive/false-negative behaviour, and whether the tests actually pin the behaviours the docstrings claim. Also read docs/plans/recursive-meta-improvement.md "Invariants" and check each of the 11 invariants against the code — report any invariant the code does not actually enforce.
I’ll review the full branch diff, check the 11 documented invariants against the scripts, and run the required checks plus targeted reproductions for any findings.
The required checks pass: 147 tests and no Ruff findings. I’m now testing gaps around the wait gate, evidence completeness, and output guards; passing tests alone don’t establish the documented invariants.
I reproduced two ways around the wait gate: changing only policy metadata, and restoring v1 through a rollback. I also confirmed that the failure miner can discard a keyword from a long error output and report a known zero count for that topic. I’m checking the remaining ancestry and write-guard cases before finalizing the findings.
The working tree changed during the review, including a fix to the rollback wait gate. I’m isolating committed HEAD (b024ff8f) and rerunning verification there so the final findings consistently describe the requested branch diff. The new uncommitted edits will remain untouched.
[P1] Policy metadata can disable evaluation gates. scripts/improvement_policy.py:93 and scripts/revise-improvement-policy.py:617
Validation does not check origin, ancestry, or consistency with history, while rollback and waiting depend on those fields. The policy hash excludes them.
Reproduction: Using identical archive, history, measurement, and v2 policy contents, changed only origin from "revision" to "init". The actual revision CLI returned exit 0 in both cases, but changed its decision from none to revise with zero qualifying rounds. An AI editing only its policy can bypass invariant 5 and rollback enforcement.
[P1] A topic name containing @ permits acceptance of a validity regression. scripts/revise-improvement-policy.py:452
Policy validation allows @, and measurement correctly resolves an exact matching definition. However, candidate_anchor() splits evidence names at @ and silently excludes that policy topic from acceptance comparisons.
Reproduction: Renamed shell-semantics to shell-semantics@custom in the four-topic test fixture and its evidence definitions; used weights [0.5, 1, 1, 1] and field counts [1, 2, 1, 0]. The decision accepted restoring the first weight, reporting validity 1.0. Remeasuring the accepted policy showed validity falling 0.5 → 0.3333. This breaks invariants 3 and 7.
[P2] Truncated failure excerpts become authoritative zero evidence. scripts/mine-trace-failures.py:199 and scripts/mine-trace-failures.py:245
Topic matching searches only the shortened command and excerpt. Discarded output is neither searched nor marked truncated, so relevant failures can become evidence of absence.
Reproduction: Supplied a status: "error" result containing a traceback header, a 200-character frame line, and ValueError: secret exposed. Mining and evidence construction returned {'secrets': 0, 'tracebacks': 1} with truncated: []. A real secret-related failure therefore appears uncorroborated. This breaks invariant 2; the tests cover listing truncation, not this content truncation.
[P2] Rolling back to v1 skips the waiting period. scripts/revise-improvement-policy.py:693
The wait gate requires base_version is not None. A rollback restoring v1 has no remaining ancestor because judged_from() returns v1’s parent, None.
Reproduction: Constructed v3 through new_version(..., origin="rollback", restored_version=1) with complete history and two unclassified archive findings. rounds_under() returned 0, but decide() returned revise. This breaks invariant 5 without malformed metadata.
[P2] Ancestor comparisons check coverage only against the immediate parent. scripts/revise-improvement-policy.py:631
The ancestor loop selects by validity alone. Without an anchor, a coverage regression inherited from a grandparent is invisible when current and parent coverage are equal.
Reproduction: Built a valid lineage where v1 recognizes quartz, v2 removes that topic, and v3 adds an unrelated topic. Two rounds stamped with v3 gave coverage 1.0, 0.0, 0.0 respectively. Although ancestry traversal returned [2, 1], the decision was revise, not rollback to v1. This breaks invariant 6; the existing multi-ancestor regression test exercises validity only.
[P2] Hard-linked outputs bypass input protection and overwrite the archive. scripts/improvement_policy.py:253
Comparing resolved paths detects symlinks but does not detect different paths sharing an inode.
Reproduction: Created temporary archive.jsonl and a hard link named report.json, then ran the measurement CLI with the archive as input and --out-json report.json. It returned 0 and replaced the archive contents with measurement JSON. This breaks invariant 9. The reproduction touched only temporary files.
Reviewed committed HEAD b024ff8f4c4e7f9356ea31258a8d954063a15675 against origin/main.
git diff origin/main...HEAD, followed by focused file/diff reads.python3 -m pytest scripts/ -q -p no:cacheprovider: 147 passed.ruff check scripts/: All checks passed!python3 - <<'PY' ... PY reproductions for all six findings. Exact consolidated reproductions: l5-review-repros.py.git archive HEAD snapshot: 147 passed, Ruff passed, and all six findings reproduced. Uncommitted changes were excluded.Invariant coverage: Findings above identify failures in 2, 3, 5, 6, 7, and 9. I found no additional defect in 1 (evidence-window alignment), 4 (literal policy-hash/archive-digest checks), 8 (bounded, unique mining), 10 (HTML escaping), or 11 (proposal workflow, inspected statically). I found no additional issue in clean-round stamping, ordinary definition variants, explicit truncation/undated-evidence handling, or role-specific destination checks. Workflow execution and live Traces behavior were not tested.
The committed branch does not hold its stated invariants. Passing tests miss reproducible evaluation bypasses, acceptance of a validity regression, and archive corruption through an output alias.