Gagan Arora shared

Public

Reviewed GitHub Actions workflow securityPublic

Agent

Codex

Model

GPT 6 Astra

Started

Sep 14, 4:31 PM

Messages28

Downloads0

Workspace

Repo

gagan114662/open-inspect-72e95a

Summary

In Progress

Understand
Verified workflow checkout behavior.
Understand
Understand
Analyzed comment trust boundaries and issue deduplication.
Understand
Understand
Identified risks in comment authentication and concurrency.
Understand

Review this new GitHub Actions workflow and its two supporting Python scripts for security and correctness bugs. This workflow is meant to run with NO secrets at all — it only reads already-posted (already-redacted) PR review comments and a repo-local archive file, then decides whether to open a GitHub issue. It must never be able to merge, deploy, or exfiltrate anything.

Focus specifically on:

Trust boundary: does checkout/execution ever run PR-controlled code with elevated permissions? (workflow_run should check out the default branch, not the PR head — verify this claim is actually true of how workflow_run behaves, and that nothing in this file overrides that default.)
Could a malicious PR author craft a PR comment (or otherwise) that causes this workflow to create issues, spam, or perform any other unintended action, given it has issues:write permission and parses comment text?
Logic bugs in the idempotency check for issue creation (race conditions, title collisions, pagination).
Bugs in the two Python scripts' threshold-crossing logic.

Output findings in the format: N. [P1|P2] description (one line each). If nothing found, say so plainly. ---WORKFLOW--- name: Archive and Recommend

Closes the specific gap named in docs/production-hardening-backlog.md

item #4's self-improvement work: scripts/detect-recurring-pattern.py could

already derive a target-vs-mechanism recommendation from

docs/self-improvement-archive.jsonl's accumulated evidence, and

scripts/analyze-latest-review.py could tell whether the LATEST round is

the one that newly tips a topic over the mechanism-fix threshold -- but

something still had to run those tools and act on the result. That "when

to act" decision was a human/agent judgment call made by reading the

archive. This workflow makes it automatic, but ONLY for a deliberately

low-risk, reversible action: opening a tracking issue. It has no

permission to merge, deploy, or touch secrets, and requests none.

Runs after "Codex Review" (.github/workflows/codex-review.yml) completes.

Deliberately uses `workflow_run`, not `pull_request` or `workflow_run`'s

head ref: workflow_run always executes the workflow file AND checks out

source from the repository's default branch, never the PR's own commits

-- so, unlike codex-review.yml, this workflow has no PR-authored-script

trust boundary to manage. It also requests no secrets at all: everything

it reads (the posted review comment, the archive file) is already public,

already-redacted PR content.

on: workflow_run: workflows: ["Codex Review"] types: [completed]

permissions: contents: read issues: write pull-requests: read

concurrency: group: archive-and-recommend-${{ github.event.workflow_run.id }} cancel-in-progress: false

jobs: analyze: name: Analyze review for recurring patterns runs-on: ubuntu-latest if: github.event.workflow_run.pull_requests[0] != null steps: - name: Checkout (default branch — trusted) uses: actions/checkout@v4 with: persist-credentials: false

  - name: Set up Python    uses: actions/setup-python@v5    with:      python-version: "3.12"
  - name: Fetch latest Codex review comment for the PR    id: fetch-comment    uses: actions/github-script@v7    with:      script: |        const prNumber = context.payload.workflow_run.pull_requests[0].number;        const comments = await github.paginate(github.rest.issues.listComments, {          owner: context.repo.owner,          repo: context.repo.repo,          issue_number: prNumber,          per_page: 100,        });        const reviewComments = comments.filter((c) =>          (c.body || "").startsWith("### Codex independent review")        );        if (reviewComments.length === 0) {          core.setOutput("found", "false");          return;        }        const latest = reviewComments[reviewComments.length - 1];        const fs = require("fs");        fs.writeFileSync(process.env.RUNNER_TEMP + "/review-comment.txt", latest.body, "utf8");        core.setOutput("found", "true");        core.setOutput("pr-number", String(prNumber));
  - name: Analyze for newly-crossed recurrence thresholds    id: analyze    if: steps.fetch-comment.outputs.found == 'true'    run: |      python3 scripts/analyze-latest-review.py \        docs/self-improvement-archive.jsonl \        "$RUNNER_TEMP/review-comment.txt" \        > "$RUNNER_TEMP/analysis.txt"      cat "$RUNNER_TEMP/analysis.txt"      # Extract just the JSON payload after the `---` separator.      awk '/^---$/{found=1; next} found' "$RUNNER_TEMP/analysis.txt" > "$RUNNER_TEMP/analysis.json"      cat "$RUNNER_TEMP/analysis.json"
  - name: Open a tracking issue for each newly-crossed topic    if: steps.fetch-comment.outputs.found == 'true'    uses: actions/github-script@v7    with:      script: |        const fs = require("fs");        const prNumber = "${{ steps.fetch-comment.outputs.pr-number }}";        const raw = fs.readFileSync(process.env.RUNNER_TEMP + "/analysis.json", "utf8");        const { newly_crossed: newlyCrossed } = JSON.parse(raw);
        if (!newlyCrossed || newlyCrossed.length === 0) {          console.log("No topic newly crossed the mechanism-fix threshold. Nothing to do.");          return;        }
        for (const item of newlyCrossed) {          const title = `Recurring pattern: ${item.topic} — mechanism-level fix recommended`;
          // Idempotency: don't open a second issue for a topic that          // already has an open tracking issue.          const existing = await github.paginate(github.rest.issues.listForRepo, {            owner: context.repo.owner,            repo: context.repo.repo,            state: "open",            labels: "self-improvement-recommendation",            per_page: 100,          });          if (existing.some((issue) => issue.title === title)) {            console.log(`Issue already open for topic "${item.topic}" — skipping.`);            continue;          }
          const body = [            `Automatically opened by [\`archive-and-recommend.yml\`](../blob/main/.github/workflows/archive-and-recommend.yml).`,            ``,            `A finding topic newly crossed the mechanism-level-fix recurrence`,            `threshold in \`docs/self-improvement-archive.jsonl\`, as of the`,            `Codex review on PR #${prNumber}.`,            ``,            `**Topic:** \`${item.topic}\``,            `**Recommended action:** ${item.recommended_action}-level fix`,            ``,            `Run \`python3 scripts/detect-recurring-pattern.py docs/self-improvement-archive.jsonl\``,            `for the full evidence trail (which rounds, which findings) behind this`,            `recommendation.`,            ``,            `This issue was opened automatically. Deciding what the mechanism-level`,            `fix should be, and merging/deploying it, remains a human decision —`,            `this workflow's only permission is \`issues: write\`.`,          ].join("\n");
          await github.rest.issues.create({            owner: context.repo.owner,            repo: context.repo.repo,            title,            body,            labels: ["self-improvement-recommendation"],          });          console.log(`Opened tracking issue for topic "${item.topic}".`);        }

---analyze-latest-review.py--- #!/usr/bin/env python3 """Decide, from evidence, whether a just-completed review round newly crosses the mechanism-fix threshold for any finding topic.

This is the piece that closes the gap named while building this archive: scripts/detect-recurring-pattern.py could already derive a target-vs- mechanism recommendation from the archive's accumulated data, but something still had to run it and decide whether the result was worth acting on -- that was a human/agent judgment call, made by eyeballing the tool's output.

This script makes that specific decision mechanical: it compares the recommendation with vs. without the latest round's findings included, and reports only topics whose recommendation flips from "target" to "mechanism" (or newly appears at/above threshold) because of this round specifically -- not topics that already crossed the threshold in earlier rounds, which would otherwise fire on every single subsequent round forever. A workflow can run this automatically after every review and act (e.g. open a tracking issue) purely on its output, with no one needing to have read the archive and noticed the pattern themselves.

Usage: python3 analyze-latest-review.py <archive.jsonl> <review-comment.txt> [--threshold N]

Exits 0 always (advisory). Prints newline-delimited human-readable lines, then a --- separator, then a JSON object: {"newly_crossed": [...]}. """

from future import annotations

import argparse import importlib.util import json import sys from pathlib import Path

def _load_sibling_module(name: str, filename: str): path = Path(file).parent / filename spec = importlib.util.spec_from_file_location(name, path) assert spec is not None and spec.loader is not None module = importlib.util.module_from_spec(spec) sys.modules[name] = module spec.loader.exec_module(module) return module

detect = _load_sibling_module("detect_recurring_pattern", "detect-recurring-pattern.py") parse_findings_mod = _load_sibling_module("parse_review_findings", "parse-review-findings.py")

def load_archive(path: str) -> list[dict]: entries = [] with open(path) as f: for line in f: line = line.strip() if line: entries.append(json.loads(line)) return entries

def next_round_number(entries: list[dict]) -> int: rounds = [e.get("round", 0) for e in entries] return (max(rounds) + 1) if rounds else 1

def recommendations_by_topic(entries: list[dict], threshold: int) -> dict[str, str]: result = detect.analyze(entries, threshold) return {rec["topic"]: rec["recommended_action"] for rec in result["recommendations"]}

def find_newly_crossed_topics( archive_entries: list[dict], new_findings: list[str], threshold: int ) -> list[dict]: """Compare recommendations with vs. without the new round's findings.

Returns entries for topics that recommend "mechanism" only once the newround is included -- i.e. this round is the one that tipped it over,not a topic that already exceeded the threshold in prior rounds."""before = recommendations_by_topic(archive_entries, threshold)
new_round_entry = {"round": next_round_number(archive_entries), "findings": new_findings}after_entries = [*archive_entries, new_round_entry]after = recommendations_by_topic(after_entries, threshold)
newly_crossed = []for topic, action in after.items():    if action != "mechanism":        continue    was_mechanism_before = before.get(topic) == "mechanism"    if not was_mechanism_before:        newly_crossed.append({"topic": topic, "recommended_action": action})
return newly_crossed

def main(argv: list[str]) -> int: parser = argparse.ArgumentParser(description=doc) parser.add_argument("archive_path") parser.add_argument("review_comment_path") parser.add_argument("--threshold", type=int, default=detect.DEFAULT_THRESHOLD) args = parser.parse_args(argv[1:])

archive_entries = load_archive(args.archive_path)
with open(args.review_comment_path) as f:    comment_text = f.read()new_findings = parse_findings_mod.parse_findings(comment_text)
if not new_findings:    print("No findings in the latest review — nothing to analyze.")    print("---")    print(json.dumps({"newly_crossed": []}, indent=2))    return 0
newly_crossed = find_newly_crossed_topics(archive_entries, new_findings, args.threshold)
if newly_crossed:    for item in newly_crossed:        print(            f"[{item['topic']}] newly recommends a MECHANISM-LEVEL fix "            f"as of this round's findings."        )else:    print("No topic newly crosses the mechanism-fix threshold this round.")
print("---")print(json.dumps({"newly_crossed": newly_crossed}, indent=2))return 0

if name == "main": sys.exit(main(sys.argv)) ---detect-recurring-pattern.py (analyze function)--- #!/usr/bin/env python3 """Decide target-fix vs. mechanism-fix from the improvement archive itself.

Closes the specific gap named while building docs/self-improvement-archive.jsonl: across rounds 1-5, I (the agent) judged when a finding was serious/recurring enough to warrant revising the improvement mechanism itself (round 5) rather than just patching the current target again (rounds 2-4). That judgment call was mine, not something derived from the archive's own data — which is exactly the gap the paper's L5 definition ("persistently revises a mechanism that governs subsequent improvement") requires closing: the recursion has to include deciding when to recurse on the mechanism, not just executing that decision once someone notices a pattern.

This script makes that decision algorithmically instead: it reads the archive, buckets every finding into a topic by keyword co-occurrence (no ML, no external calls -- deliberately simple and auditable), and recommends "mechanism" once a topic has recurred at or above a threshold across distinct rounds, "target" otherwise. It is still a human (or an agent executing on the human's behalf) who reads the recommendation and acts on it -- this does not make merge/deploy autonomous, and does not claim to. What it closes is narrower and real: the decision itself is now reproducible from evidence, not from an agent's unrecorded judgment.

Usage: python3 detect-recurring-pattern.py <archive.jsonl> [--threshold N]

Prints one recommendation line per topic that has reached the threshold, plus a machine-readable JSON summary to stdout after a --- separator. Exit code 0 always (this is advisory, not a pass/fail gate). """

from future import annotations

import argparse import json import sys from collections import defaultdict

DEFAULT_THRESHOLD = 3

Deliberately simple, auditable keyword buckets rather than an embedding

model or external call -- the point of this tool is that its decision

process is itself inspectable, matching the same "no unexplained black

box" standard the rest of this archive holds code changes to.

TOPIC_KEYWORDS: dict[str, list[str]] = { "credential-redaction": ["redact", "credential", "secret", "token", "leak", "expos"], "shell-semantics": ["errexit", "bash -e", "exit code", "-e", "pipefail", "shell"], "env-var-precedence": ["precedence", "env var", "environment variable", "unconditionally"], "fork-pr-permissions": ["fork", "github_token", "persist-credentials"], "auth-lifecycle": ["refresh token", "rotat", "expir", "auth.json", "stale"], }

def classify_finding(text: str) -> str | None: lowered = text.lower() for topic, keywords in TOPIC_KEYWORDS.items(): if any(kw in lowered for kw in keywords): return topic return None

def load_archive(path: str) -> list[dict]: entries = [] with open(path) as f: for line in f: line = line.strip() if line: entries.append(json.loads(line)) return entries

def analyze(entries: list[dict], threshold: int) -> dict: topic_rounds: dict[str, set[int]] = defaultdict(set) topic_examples: dict[str, list[str]] = defaultdict(list)

for entry in entries:    round_num = entry.get("round")    for finding in entry.get("findings", []):        topic = classify_finding(finding)        if topic is None:            continue        topic_rounds[topic].add(round_num)        if len(topic_examples[topic]) < 3:            topic_examples[topic].append(f"round {round_num}: {finding[:120]}")
recommendations = []for topic, rounds in sorted(topic_rounds.items(), key=lambda kv: -len(kv[1])):    recurrence = len(rounds)    action = "mechanism" if recurrence >= threshold else "target"    recommendations.append(        {            "topic": topic,            "recurrence_count": recurrence,            "rounds": sorted(rounds),            "recommended_action": action,            "examples": topic_examples[topic],        }    )
return {"threshold": threshold, "recommendations": recommendations}

def main(argv: list[str]) -> int: parser = argparse.ArgumentParser(description=doc) parser.add_argument("archive_path") parser.add_argument("--threshold", type=int, default=DEFAULT_THRESHOLD) args = parser.parse_args(argv[1:])

entries = load_archive(args.archive_path)result = analyze(entries, args.threshold)
for rec in result["recommendations"]:    marker = "MECHANISM-LEVEL FIX RECOMMENDED" if rec["recommended_action"] == "mechanism" else "target-level fix sufficient so far"    print(        f"[{rec['topic']}] recurred in {rec['recurrence_count']} round(s) "        f"{rec['rounds']} -> {marker}"    )    for example in rec["examples"]:        print(f"    - {example}")
print("---")print(json.dumps(result, indent=2))return 0

if name == "main": sys.exit(main(sys.argv))

I’ll verify GitHub’s workflow_run checkout behavior and review the comment trust boundary, issue deduplication, and threshold logic. I’m using the agent-reach skill for the documentation lookup.

The checkout claim holds for this file: workflow_run supplies the default-branch ref and SHA, and checkout has no override. I found separate problems with accepting unauthenticated comments, selecting reviews unrelated to the completed run, and allowing concurrent issue creation. I’m checking the threshold logic with small examples.

Checkout is safe as written: workflow_run supplies the default-branch ref/SHA, which checkout uses without a ref override. Both API queries paginate; no comment-to-code execution or merge/deploy path was found.

[P2] The comment filter authenticates only a header: any PR commenter can impersonate Codex and supply numbered findings that trigger issues for topics one round below threshold, or suppress recommendations with an empty forged review.
[P2] The selected comment is not tied to the triggering run, attempt, or commit; failed/cancelled runs that post nothing reuse older findings, while overlapping runs can consume another run’s review.
[P2] Concurrency is scoped to individual run IDs, so different runs can simultaneously pass the open-issue check and create duplicate issues; the check/create sequence needs repository-wide serialization.
[P2] No processed-round state is retained: with an unchanged archive, subsequent reviews repeatedly report the same crossing and recreate closed tracking issues, while topics needing multiple additional rounds never accumulate enough evidence.
[P2] find_newly_crossed_topics() always assigns max(round)+1 without checking review identity; replaying a review already represented in a two-round archive fabricates a third distinct round and falsely crosses the threshold.

Summary

Closes the specific gap named in docs/production-hardening-backlog.md

item #4's self-improvement work: scripts/detect-recurring-pattern.py could

already derive a target-vs-mechanism recommendation from

docs/self-improvement-archive.jsonl's accumulated evidence, and

scripts/analyze-latest-review.py could tell whether the LATEST round is

the one that newly tips a topic over the mechanism-fix threshold -- but

something still had to run those tools and act on the result. That "when

to act" decision was a human/agent judgment call made by reading the

archive. This workflow makes it automatic, but ONLY for a deliberately

low-risk, reversible action: opening a tracking issue. It has no

permission to merge, deploy, or touch secrets, and requests none.

Runs after "Codex Review" (.github/workflows/codex-review.yml) completes.

Deliberately uses workflow_run, not pull_request or workflow_run's

head ref: workflow_run always executes the workflow file AND checks out

source from the repository's default branch, never the PR's own commits

-- so, unlike codex-review.yml, this workflow has no PR-authored-script

trust boundary to manage. It also requests no secrets at all: everything

it reads (the posted review comment, the archive file) is already public,

already-redacted PR content.

Deliberately simple, auditable keyword buckets rather than an embedding

model or external call -- the point of this tool is that its decision

process is itself inspectable, matching the same "no unexplained black

box" standard the rest of this archive holds code changes to.

Summary

Closes the specific gap named in docs/production-hardening-backlog.md

item #4's self-improvement work: scripts/detect-recurring-pattern.py could

already derive a target-vs-mechanism recommendation from

docs/self-improvement-archive.jsonl's accumulated evidence, and

scripts/analyze-latest-review.py could tell whether the LATEST round is

the one that newly tips a topic over the mechanism-fix threshold -- but

something still had to run those tools and act on the result. That "when

to act" decision was a human/agent judgment call made by reading the

archive. This workflow makes it automatic, but ONLY for a deliberately

low-risk, reversible action: opening a tracking issue. It has no

permission to merge, deploy, or touch secrets, and requests none.

Runs after "Codex Review" (.github/workflows/codex-review.yml) completes.

Deliberately uses workflow_run, not pull_request or workflow_run's

head ref: workflow_run always executes the workflow file AND checks out

source from the repository's default branch, never the PR's own commits

-- so, unlike codex-review.yml, this workflow has no PR-authored-script

trust boundary to manage. It also requests no secrets at all: everything

it reads (the posted review comment, the archive file) is already public,

already-redacted PR content.

Deliberately simple, auditable keyword buckets rather than an embedding

model or external call -- the point of this tool is that its decision

process is itself inspectable, matching the same "no unexplained black

box" standard the rest of this archive holds code changes to.

Deliberately uses `workflow_run`, not `pull_request` or `workflow_run`'s

Deliberately uses `workflow_run`, not `pull_request` or `workflow_run`'s