Reviewed L5 meta-improvement branch
Public
Reviewed L5 meta-improvement branch
Agent
Codex
Model
GPT 6 Astra
Started
Sep 14, 11:50 PM
Messages44
Downloads0
Workspace
Reviewed full branch diff and scripts.
UnderstandRan tests and workflow checks.
VerifyConfirmed archive-integrity issues with SHA markers and round numbers.
ReproduceYou are an independent reviewer with no stake in prior conclusions. Review the ENTIRE branch feat/l5-meta-improvement against origin/main (run git diff origin/main...HEAD — do NOT limit yourself to the latest commit). This branch adds an "L5" meta-improvement layer: a versioned improvement policy (docs/improvement-policy.json), scripts under scripts/ (improvement_policy.py, detect-recurring-pattern.py, archive-round.py, measure-policy-validity.py, revise-improvement-policy.py, mine-trace-failures.py, render-rsi-dashboard.py and their _test.py), workflows under .github/workflows/ (codex-review.yml, revise-improvement-policy.yml, archive-and-recommend.yml), and docs (docs/plans/recursive-meta-improvement.md, docs/rsi/).
Threat model: the meta-improver (an AI) may only rewrite the policy it owns; it must never be able to widen its own authority, game its own evaluation (coverage/validity), corrupt the archive or verifier, or accept a revision on evidence that does not support it. Findings should be things that are actually wrong, with a concrete reproduction. Run commands to verify: python3 -m pytest scripts/ -q -p no:cacheprovider, ruff check scripts/, and targeted python3 - <<'PY' ... PY reproductions. Never claim a result you did not observe.
Output format: a numbered list of findings, each tagged [P1] (must fix), [P2] (should fix) or [P3] (nit), each with file:line, the failure scenario, and how you reproduced it; then a "## Verification" section listing the commands you ran and their results; then a "## Verdict" line: whether the branch, as a whole, holds its stated invariants. If you find nothing in an area, say so explicitly.
SCOPE B — the GitHub Actions workflows on this branch: .github/workflows/codex-review.yml, revise-improvement-policy.yml, archive-and-recommend.yml (and any other workflow the diff touches). Check: trust boundaries (what a PR author can influence), where artifacts live, the codex-review-status stamp and whether the review text can forge it, secret handling, the proposal-branch/PR flow, idempotency, whether the archive round SHA logic can drop or duplicate rounds, whether an attacker PR could make the meta-improver write outside its allowed files, and whether the workflows call the scripts with flags the scripts actually define (check every python3 scripts/... invocation against the script's argparse). You cannot run Actions; reason from the YAML and reproduce script invocations locally where possible.
I’ll review the full branch diff, focus on workflow trust boundaries and script invocations, and run the requested checks plus targeted reproductions.
Ruff passed; pytest is still running. The workflow flags I’ve checked match the scripts’ parsers. I’m now checking whether review output and SHA markers reliably bind archived evidence to the review that produced it.
All 147 tests passed. I reproduced two concurrent archive proposals receiving the same round number; I’m checking how that affects L5’s evaluation window. I’m also using the agent-reach skill’s web-reading route to verify GitHub’s rules for reviews triggered by bot-created PRs.
The reproductions confirmed two archive-integrity problems: review text can supply a SHA marker that makes a newer comment match an older run, and two distinct review SHAs can collapse into one L5 round. The status-marker sanitization itself worked. All workflow script invocations I exercised succeeded in an isolated copy.
GitHub’s current documentation says bot-created PRs require approval before their review workflows run; I’m not treating that approval gate as a defect.
[P1] Review text can forge the SHA binding and corrupt archived evidence.
Files: codex-review.yml:346, archive-and-recommend.yml:134.
The posting step sanitizes codex-review-status, but leaves codex-review-sha untouched. The archive workflow accepts a matching SHA marker anywhere in a bot-authored comment and selects the latest match.
If review B quotes or reproduces <!-- codex-review-sha: SHA_A --> while A’s archive job is pending, that job can select B’s findings—or B’s clean result—as evidence for A. The archive then consumes A’s SHA, preventing correction on retry.
Reproduction: Ran the actual JavaScript sanitizer and equivalent selection predicate through Node, using a B comment containing A’s marker. Observed wrongShaMatches: 1. Passed the resulting comment to archive-round.py: it exited 0 and recorded source_sha: aaaa… with empty findings. The forged status marker was correctly neutralized; the SHA marker survived.
Bind comments to trusted run metadata, or validate one authoritative footer and sanitize SHA markers in review text too.
[P2] Concurrent archive proposals collapse distinct reviews into one evaluation round.
Files: archive-and-recommend.yml:158, measure-policy-validity.py:109, revise-improvement-policy.py:538.
Two archive proposals created before either merges compute the same next integer round from main. Keeping both appended records when merging does not resolve that collision. The new L5 measurement combines them by round number, and rounds_under() counts them once despite their different SHAs. This delays evaluation/rollback and combines separate evidence epochs.
Reproduction: Called build_round_entry() twice against the checked-in archive with distinct SHAs. Both received round 11. Combining both records increased the measured round count by 1, and rounds_under([a, b], policy) returned 1.
Use a stable unique review identity for automated rounds, retaining separate handling for legacy placeholder/result pairs.
git diff origin/main...HEAD — captured the entire branch diff; inspected all three changed workflows and their script interfaces.python3 -m pytest scripts/ -q -p no:cacheprovider — 147 passed.ruff check scripts/ — All checks passed.python3 - <<'PY' … PY reproductions — confirmed both findings. One initial attempt referenced a nonexistent helper; corrected to rounds_in_order() and reran successfully.No—the branch as a whole does not hold its stated invariants: workflow evidence can be bound to the wrong SHA, and distinct reviews can collapse into one L5 evaluation round.