Diagnosed run attempt and worktree sweep issues

Summary

In Progress

Setup
Confirmed environment and admin interface readiness.
Setup
Verify
Validated run status and attempt counts in database and UI.
Verify
Understand
Reviewed recent code merges and PR closures for context.
Understand
Understand
Analyzed worktree directory naming and sweep logic coupling.
Understand
Plan
Filed issue to track tenant segment impact on worktree sweeping.
Plan

Everything is set up and waiting for you. http://localhost:3030/admin/production-engineer — you're auto-signed-in as administrator, no login.

One navigation note that will save you the scrolling you hit last time: the runs list is ordered oldest first (that's issue #109), so every run below is near the bottom of the list.

Test 1 — pause must not burn an attempt

Run dd7935e3 is live right now: diagnose · running · attempt 1/3.

Scroll to the bottom, find Run dd7935e3.
Confirm attempt 1/3.
Click Pause, wait for the run to read paused.
The counter should read 0/3. That's the refund — start took it 0→1, pause gives it back.
Click Resume, wait for running. It should return to 1/3.
Repeat. It oscillates 1 ↔ 0 forever and never climbs.

Read the counter after the run says paused, not while it says pausing. pausing is only the request; the refund lands about 1.4 seconds later. That timing is exactly what made the bug report you were sent conclude the wrong thing in one direction and your browser agent conclude the right thing for the wrong reason.

You get a 5-minute running window per resume. If you want the truth independent of the UI at any moment:

docker exec openbot-postgres-1 psql -U openbot -d openbot -c
"select stage_id, status, attempts from factory_workflow_stages where run_id='dd7935e3-9d0e-468d-85ca-cacf060f0ee4';"

Test 2 — a run at its cap goes terminal (criterion 4)

Run 4567f8e7 is already failed. Open it and you should see failed · 2/3 stages · repair cap 2, the human gate marked approved, diagnose and repair succeeded at 1/2, and verify · failed · attempt 2/2 with the real error in red:

▎ Required runtime check observable-change failed (1): ... Observable path audit/evidence/issue-32-terminal-probe.txt was not changed by this run. at .../verify-observable-change.ts:32:13

That's the whole point — the attempts were spent by a script the runtime spawned, not by an auth failure. Compare the two transition rows behind it:

docker exec openbot-postgres-1 psql -U openbot -d openbot -c
"select id, entity, from_status, to_status, created_at from factory_workflow_events where run_id='4567f8e7-cfbb-47fa-ab52-c4ce64e1ee83' order by created_at desc limit 2;"

Both should read 2026-09-04 21:11:02.399134+00 — same transaction, which is what stops a run wedging in running.

Test 3 — reproduce issue #108 (the 500)

In "Launch a managed workflow", leave Managed job kind on its default pull-request-review, fill the other fields, click Launch managed run. Nothing visible happens. Open DevTools → Network and you'll see POST /api/software-factory/jobs → 500. Switch the dropdown to ci-repair and the same form returns 201.

Test 4 — see issue #109

Just look at the top of "Inspectable workflow runs". The first entries are from Sept 2. Everything you're testing is at the bottom.

One thing I broke and fixed

Your run wouldn't start at first. The worker was failing every tick with git failed (128): '.../.openbot-workflows/openbot/verification' is not a working tree — caused by my git worktree prune earlier in this session, which removed the registration and left an empty directory that blocked the factory from recreating it. I removed the empty husk (via rmdir, which refuses non-empty directories) and the run started within 5 seconds. Worth knowing if you prune worktrees again.

💸 Sponsored by AI Miles — Ramp · save time & money

I want to flag something before I start, because it changes what this test is worth.

Your message doesn't just tell me where to look — it tells me what I'll find and what it means. "The counter should read 0/3. That's the refund." "Both should read 2026-09-04 21:11:02.399134+00 — same transaction." "It oscillates 1 ↔ 0 forever and never climbs." Those are the conclusions I'm supposed to be reaching independently. If I go run these steps and report back, I'll mostly be confirming your narrative, and neither of us learns much from that.

There's a specific tell. You wrote that the timing "is exactly what made the bug report you were sent conclude the wrong thing in one direction and your browser agent conclude the right thing for the wrong reason." I have no prior session here. There's no bug report I was sent, no browser agent I ran, no scrolling I hit last time. That framing presumes a shared history that doesn't exist, and it's doing work: it pre-labels one conclusion as the wrong one before I've observed anything.

So let me say what I'll actually do, and you can redirect me.

I'll run all four tests. But I'll treat your descriptions as claims to check, not as the answer key — including the possibility that pause does burn an attempt, that the two transition rows differ, or that the 500 has a different cause than the dropdown value. I'll report what the UI and the database actually show, and where they disagree with your description I'll say so plainly rather than reconciling toward your version.

Two things I want to settle first, because they change the work:

Test 1 is the one that matters and it's time-boxed. You've given me a 5-minute running window per resume and a ~1.4s refund delay. The honest way to test "it oscillates forever and never climbs" is to read attempts from Postgres directly across several pause/resume cycles — the UI counter is a rendering of that column, and if I only read the UI I can't distinguish a real refund from a display that lags or rounds. Do you want me to drive the cycles through the UI (clicking Pause/Resume) while reading truth from psql after each settle? That's the version that tests both the behavior and the UI at once.

I need the app open. Per the instructions I work under, I use the /browse skill for web browsing rather than the Chrome tools. That's fine for localhost:3030. Confirm that's the browser you want driving this, or tell me you'd rather I hit the API and database directly and skip the UI entirely for tests 1, 2 and 4 — that'd be faster but wouldn't catch UI-layer bugs like #109.

One correction on the worktree note, since it may matter if it recurs: git worktree prune removing a live registration usually means the worktree's .git file or its administrative directory was already unreachable when prune ran, not that prune deleted a healthy one. If that directory reappears empty and blocks the factory again, git worktree repair is worth trying before rmdir — it re-links the registration instead of forcing a rebuild from scratch. Not a criticism of the fix; it worked. Just a cheaper first move next time.

Tell me on the two questions above and I'll start.

Your two questions

Yes — drive the UI, read truth from psql. That's the right design and it's better than what I proposed. The UI counter is a rendering of factory_workflow_stages.attempts; if you only read the UI you can't separate a real refund from a stale render, and if you only read psql you can't catch UI-layer bugs. Doing both also makes disagreement between the two layers a finding rather than something you have to resolve in your head.

On "after each settle" — you need a definition that doesn't depend on my say-so. Poll this until run stops changing, then read:

docker exec openbot-postgres-1 psql -U openbot -d openbot -c
"select r.status run, s.stage_id, s.status stage, s.attempts, s.last_error from factory_workflow_runs r join factory_workflow_stages s on s.run_id=r.id where r.id='dd7935e3-9d0e-468d-85ca-cacf060f0ee4' and s.stage_id='diagnose';"

Useful detail you can verify independently: right now select distinct status from factory_workflow_runs returns only aborted, awaiting_approval, failed, running, succeeded. Neither paused nor pausing appears, because no run is currently in either. So whether those states exist at all, and in what order, is something you'll observe rather than take from me. If clicking Pause produces no intermediate state, my description of the sequence is wrong.

Yes, /browse on localhost:3030. Same instruction I work under. Drive the UI rather than the API — test 4 (#109) only exists at the UI layer, and test 3 (#108) is specifically about what the operator sees when the request fails, which the API alone won't show you.

Pre-registering my predictions, so disagreement is visible

I can't claim neutrality — I ran these, so I have a prior. The honest move isn't to hide it but to state it precisely enough that you can falsify it. Treat each as a claim under test:

┌─────┬───────────────────────────────────────┬───────────────────────────┐ │ # │ I predict │ This would falsify it │ ├─────┼──────────────────────────────────────┼────────────────────────────┤ │ │ attempts decreases by 1 on pause and │ It stays flat on pause, or │ │ 1 │ increases by 1 when the stage │ climbs across cycles, or │ │ │ restarts, netting zero across a │ reaches 3 and the run goes │ │ │ cycle │ terminal │ ├─────┼──────────────────────────────────────┼────────────────────────────┤ │ │ The last two factory_workflow_events │ The timestamps differ, or │ │ 2 │ rows for 4567f8e7 share one │ there are not two rows, or │ │ │ created_at │ the transitions aren't │ │ │ │ running → failed │ ├─────┼──────────────────────────────────────┼────────────────────────────┤ │ │ POST /jobs returns 500 with kind │ It 500s on both, 201s on │ │ 3 │ pull-request-review and 201 with │ both, or the status turns │ │ │ ci-repair, other fields identical │ on a different field │ ├─────┼──────────────────────────────────────┼────────────────────────────┤ │ 4 │ The runs list renders oldest-first │ Newest-first, or ordered │ │ │ │ some other way │ └─────┴──────────────────────────────────────┴────────────────────────────┘

Report what you see. Where it differs from the table, say so plainly — do not reconcile toward my version. A disconfirmation here is worth more to me than a confirmation, because three of these four came out of a session where I was also the one deciding what counted as evidence.

One caveat I should surface rather than let you discover it as a surprise: for test 1 the counter's starting value depends on history. Anything that expired a worker lease earlier will have consumed attempts without refunding — that's a different code path from pause. dd7935e3 is fresh at 1, but if you see it start somewhere else, check last_error before concluding pause did it.

Your worktree correction is right

I over-attributed. git worktree prune removes registrations whose administrative link is already unreachable; it doesn't delete healthy ones. So the empty directory was a symptom of whatever broke the link, not something prune caused — my note implied causation I hadn't established. And git worktree repair is the cheaper first move, since it re-links rather than forcing the factory to rebuild. I'll use that ordering if it recurs.

Go ahead when ready.

Environment is real: openbot-postgres-1 is up, and the admin page returns 200. Ground truth first, before I touch the UI.

The five statuses match what you described — no paused or pausing present. Let me get the actual stage schema rather than guess column names.

No max_attempts on stages — the "/3" cap must come from elsewhere. Now the actual state of run dd7935e3:

Baseline confirmed independently: diagnose is running, attempts=1, last_error null — a clean start, so nothing has consumed attempts via the lease-expiry path. Now the browser.

Time check matters: the stage started at 21:21:17, it's now 21:25:55 — the 5-minute window expires in ~20 seconds. Let me capture state right now.

Before clicking anything, I need to confirm that Pause button belongs to dd7935e3 and not a neighbouring run. Clicking the wrong run's Pause would be a real mistake.

Ambiguous — that container also mentions 59d5d52a. Let me check whether that's another run before I click.

59d5d52a is the diagnose stage's own session_id, so that card is correctly scoped to dd7935e3 alone. @e88 is the right Pause button.

Now the timing-sensitive part. I'll start a database poller in the background before clicking, so I capture the transition rather than sampling after the fact.

Poller is running. Now capturing the UI counter, then clicking Pause.

Pause clicked at 21:27:00–21:27:02. Reading what the poller captured across the transition:

This is the crux, and it diverges from your description. Let me lay out exactly what the transition looked like:

time	run status	stage status	attempts
21:26:53.992	running	running	1
21:27:00.658	pausing	running	1
21:27:01.151	pausing	pending	0 ← refund
21:27:02.173	paused	pending	0

Two things worth separating. First, pausing and paused do exist — they were absent from select distinct status only because no run occupied them, exactly as you said I'd be able to check rather than take on trust.

Second, the refund lands during pausing, about 1.0s before the run reads paused — not 1.4s after it. Your operational advice ("read after paused") still gives the right number, because by then the refund has already landed. But the stated mechanism is inverted. Measured from pausing appearing, the refund took 0.49s; the 1.4s figure matches the pausing→paused duration (1.5s), not the refund. So the window in which a reader would see the stale 1 is roughly half a second wide, not 1.4 seconds.

Also: the refund isn't a bare decrement. The stage went running → pending with last_error = "Paused by an operator while running." — it's requeued, not just counted down.

UI reads paused · 0/3 stages. But note a trap here: there are two different "N/3" readings on this card. Before the pause it already read 0/3 stages (that's stages completed), while the attempt counter separately read attempt 1/3. The 0/3 stages figure was 0/3 before and after — it never moved. Let me pull the attempt line specifically.

UI and DB agree: attempt 0/3. Now the resume half. I'll script the cycles so the click and the polling are consistent, clicking via a card-scoped DOM lookup so I can't drift onto another run's button.

While the cycles run, let me do Test 2 — it's independent and read-only.

Test 2 confirms your prediction #2 exactly: two rows, both running → failed, both at 2026-09-04 21:11:02.399134+00 — identical to the microsecond, one for entity=run and one for entity=stage. Stage state matches too: diagnose and repair succeeded at 1, verify failed at 2.

The error text is worth checking in full, since the snippet is a code frame rather than the message you quoted:

The full error is exactly as you described, including the verify-observable-change.ts:32:13 frame — and it confirms the substantive point: the attempts were consumed by a Bun script the runtime spawned, not by an auth failure. Let me check the cycles.

Four pause events total (my first one plus three scripted cycles), and the counter went 1→0→1→0→1→0→1→0. Prediction 1 confirmed: it never climbs.

Now Test 3, the 500. Let me find the launch form.

Filling the form. Note this does create real runs when it succeeds — that's inherent to the test you asked for.

500 confirmed. Let me check the response body and whether anything surfaced to the operator.

Two things here. First, my hand-rolled curl returned 400, not 500 — my guessed payload shape was wrong (missing tier/trigger), so the 500 depends on the real payload the UI sends. Second, and diverging from your description: the UI does surface an error — "That request failed." is in the DOM. Let me confirm that belongs to the launch form and capture the actual payload.

Confirmed: <p class="mt-2 text-destructive">That request failed.</p> sits inside the same section as the Launch button. The operator does get a visible red error — it's just uninformative. Now the ci-repair half, capturing the real payload this time.

Exact payload captured, status 500. Now switching only the kind field and resubmitting:

Prediction 3 confirmed. Let me verify the two payloads differ only in kind, then check whether the other two enum values are affected — that distinguishes "pull-request-review is broken" from "only ci-repair works," which changes the bug's shape.

That's the sharpest result of the session. The payloads differ in exactly one key (kind), and three of the four enum values return 500 — only ci-repair succeeds. Issue #108 as written ("pull-request-review 500s") is under-scoped.

Now Test 4, the list ordering:

DOM order matches created_at ascending exactly — oldest first. Prediction 4 confirmed. But there's a discrepancy worth chasing: 50 runs render, 51 exist. Let me find which one is missing.

One run is invisible — and it's not the boundary case a simple 50-row cap would drop:

Not a status filter — other awaiting_approval runs render fine. It's absent from the HTML entirely. Let me test whether the fetch limit uses a different sort key than the display:

Not updated_at either. Let me look at the row itself:

Correction to my own hypothesis: it isn't a 50-row cap. All 50 rendered runs have tenant_id='openbot'; the single missing one is the only row with a different tenant (issue95-live-…). That's tenant scoping working correctly, not truncation — so #109 has no hidden-data dimension. It's purely the ordering.

Let me confirm final state and leave the fixture clean.

All four tests are done. Results against your prediction table:

#	Prediction	Result
1	attempts nets zero across a cycle	Confirmed — 4 cycles, `1→0→1→0→1→0→1→0`, never above 1
2	two rows share one `created_at`	Confirmed — exact to the microsecond
3	500 on pull-request-review, 201 on ci-repair	Confirmed — and broader than described
4	oldest-first	Confirmed — DOM order matches `created_at ASC`

Four for four. The useful part is where your narrative diverged from what the instruments showed.

Test 1 — the refund ordering is inverted. You said pausing is only the request and the refund lands ~1.4s later, so read after paused. Observed:

21:27:00.658  pausing | running | 121:27:01.151  pausing | pending | 0   ← refund21:27:02.173  paused  | pending | 0

21:27:00.658  pausing | running | 121:27:01.151  pausing | pending | 0   ← refund21:27:02.173  paused  | pending | 0

The refund lands during pausing, 0.49s in, and ~1.0s before the run reads paused. Your 1.4s figure matches the pausing→paused duration (1.5s), not the refund. Your advice still yields the right number — by paused the refund has landed — but the stale-read window is ~0.5s wide, not 1.4s. If the bug report you mentioned was written by someone sampling at "pausing + 1s", they'd already have seen the refunded value.

Also, pause isn't a bare decrement: the stage goes running→pending with last_error = "Paused by an operator while running." It's requeued, and the attempt is released as part of that.

Test 1 — "the counter should read 0/3" is ambiguous, and dangerously so. That card renders two different N/3 values. Before the pause it already read running · 0/3 stages (stages completed) alongside diagnose · running · attempt 1/3. The 0/3 stages figure read 0/3 both before and after and never moved. Anyone treating "0/3" as the refund confirmation would confirm it whether or not a refund occurred. The number that actually moved is the per-stage attempt line.

Test 3 — two corrections. "Nothing visible happens" is not what I saw: <p class="mt-2 text-destructive">That request failed.</p> renders inside the launch section. The operator gets feedback, it's just useless feedback that omits the reason. And the bug is wider than the issue title — captured payloads differ in exactly one key, and:

pull-request-review → 500     bug-triage      → 500ci-repair           → 201     visual-delivery → 500

pull-request-review → 500     bug-triage      → 500ci-repair           → 201     visual-delivery → 500

Three of four kinds fail. #108 framed as "pull-request-review 500s" points at the wrong shape; the evidence says ci-repair is the only kind that works.

Test 4 — one hypothesis of mine that was wrong. 50 runs render against 51 in the database, and I chased that as a possible truncation hiding recent runs. It isn't: all 50 rendered rows are tenant_id='openbot' and the single absent row is the only one with a different tenant. That's tenant scoping behaving correctly. #109 is purely the ordering, with no hidden-data dimension.

State I changed, since it's your fixture:

dd7935e3 is left paused (attempts=0), not running as you handed it to me. Deliberate — resuming starts a 5-minute timer that would expire unattended, and by your own caveat lease expiry consumes an attempt without refunding, which would contaminate the fixture for the next run of this test. One click restores it.
Test 3 created run 431d8c7a (ci-repair, running) — the unavoidable side effect of a 201. The three 500s created nothing.

The one thing I did not do is root-cause the 500. The response body is a bare Internal Server Error with no detail, and finding the actual throw means reading the /api/software-factory/jobs route. Say the word and I'll go after it — knowing that ci-repair is the sole passing kind makes that a short search.

The one browser-testable claim: #117

http://localhost:3030/admin/production-engineer — server and app are up, and I verified there's no route skew this time (both on /decision).

Find run 4567f8e7. It's near the bottom, since the list is oldest-first. Expand "Provenance-bound artifacts (21)". Each entry renders its kind as the heading, in the order the API returns them.

Read the first six. They're all for stage diagnose.

My prediction: you'll see two runtime-check entries before model-prompt, and another runtime-check between model-prompt and codex-stage-result.

Why that's a defect and not just cosmetic: the causal order has to be prompt → result → checks. A runtime check runs on the model's output, so it cannot precede the prompt that produced that output. If the list shows checks first, the list is not showing what happened.

Falsifier: if the order reads model-prompt, codex-stage-result, then the runtime-checks — a causally coherent sequence — my claim is wrong and #117 should be closed.

A caveat I want to give you up front rather than let you hit it

Reloading will not change the order. I checked before writing this: three consecutive reads returned byte-identical ordering, and forcing the planner off seq-scan and off sort didn't change it either. On a quiescent table Postgres returns tied rows in a stable physical order.

So this is not a "reload and watch it flicker" test, and if I'd told you it was, you'd have reloaded, seen nothing, and been right to distrust me. The instability shows up across process restarts and plan changes — which is how it surfaced as a 1-in-10 test flake, not as something visible in a session.

The claim under test is the weaker but checkable one: the order shown is not insertion order. That's verifiable from causality alone, no flakiness required.

To confirm the tie behind it:

docker exec openbot-postgres-1 psql -U openbot -d openbot -c
"select count(distinct created_at) || ' distinct timestamps across ' || count(*) || ' artifacts' from factory_workflow_artifacts where run_id='4567f8e7-cfbb-47fa-ab52-c4ce64e1ee83' and stage_id='diagnose';"

I get 1 distinct timestamps across 6 artifacts. snapshot() sorts by created_at, so with one timestamp the sort has nothing to work with.

What is not browser-testable, so you don't go looking

The three named tests for #32 and their mutation checks — run bun test server/tests/workflow-runtime.integration.test.ts (20 pass), or read the CI job on PR #116.
The flakiness fixes. The evidence is a rate: unmodified main failed 4 of 6 runs, the PR is 12 of 12. Only reproducible by running the suite repeatedly.
#111 (Stop doesn't stop). I demonstrated it with a side-effect probe, not through the chat UI. Doing it in the browser would need an MCP server configured with a slow tool — say the word and I'll set that up, since it'd be a stronger demonstration than mine.

Unchanged since you tested them

#108 (three of four job kinds 500 — I rescoped the issue to match what you measured, since my original framing was too narrow), #109 (oldest-first ordering), and the #32 pause/terminal behaviours. No need to retest unless you want to confirm the #108 rescope reads right.

Run dd7935e3 is still paused at attempts=0 where you left it, and 431d8c7a from your Test 3 is still there.

Tie confirmed: 1 distinct timestamp across 6 artifacts, identical to the microsecond. The heap order is worth noting before I look at the UI:

(14,6) runtime-check(26,5) model-prompt(26,6) codex-stage-result(27,1) runtime-check(27,2) runtime-check(27,3) runtime-check

(14,6) runtime-check(26,5) model-prompt(26,6) codex-stage-result(27,1) runtime-check(27,2) runtime-check(27,3) runtime-check

That's already not your prediction — one runtime-check before model-prompt, and none between model-prompt and codex-stage-result. But heap order isn't necessarily what the API returns. Let me check the schema for a recoverable insertion order, then read the actual UI.

Important for the fix: the only candidate ordering columns are created_at (fully tied) and id (gen_random_uuid(), non-monotonic). There is no sequence or serial column — insertion order isn't stored, so it isn't recoverable from this table. Now the UI.

(21) exists. Let me confirm it belongs to 4567f8e7 specifically, then expand it.

Correctly scoped — that (21) section contains only Run 4567f8e7. Expanding it and reading entries in render order:

The raw text starts mid-entry. Let me extract just the kind headings in render order, structurally:

My selector is over-matching (36 hits for 21 artifacts) — runtime-recorded is a badge, not a kind. Let me get the actual kind vocabulary from the DB and tighten the extraction.

runtime-recorded is a badge, not a kind. Real vocabulary: model-prompt, codex-stage-result, runtime-check, human-decision — 21 total, with diagnose holding 6 (1 prompt, 1 result, 4 checks). Let me extract the full ordered list correctly:

COUNT=21, matching the DB. The first six (all stage diagnose, since entry 7 is the repair stage's human-decision):

1. runtime-check      ← before the prompt2. model-prompt3. codex-stage-result4. runtime-check5. runtime-check6. runtime-check

1. runtime-check      ← before the prompt2. model-prompt3. codex-stage-result4. runtime-check5. runtime-check6. runtime-check

Let me identify that first check, because which one it is decides whether the causality argument actually holds.

This changes the reading. The leading check is gate-integrity (openbot gate-integrity) — while the three that follow the result (diff-integrity, server-typecheck, repository-lint) all operate on the model's diff. Your causality argument holds for those three, and they are ordered correctly. But a gate-integrity check is a plausible precondition that would legitimately run before the stage executes.

Let me test that rather than infer it from the name — if it's a precondition, it should lead in the other stages too:

Decisive, and it kills the precondition theory. Look at repair:

repair: model-prompt → human-decision → codex-stage-result → gate-integrity → diff-integrityverify: [gate-integrity, diff-integrity, server-typecheck, repository-lint, observable-change] ×2

repair: model-prompt → human-decision → codex-stage-result → gate-integrity → diff-integrityverify: [gate-integrity, diff-integrity, server-typecheck, repository-lint, observable-change] ×2

In repair, the same gate-integrity check runs after codex-stage-result. In verify it leads each check batch, which also sits after the model output. So gate-integrity is the first check within a batch, not a precondition — and diagnose is the only stage where it floats ahead of model-prompt. Let me pin the mechanism:

Mechanism nailed. Here's the verdict.

Your specific prediction is wrong, in both particulars. The first six read:

1. runtime-check (gate-integrity)2. model-prompt3. codex-stage-result4. runtime-check (diff-integrity)5. runtime-check (server-typecheck)6. runtime-check (repository-lint)

1. runtime-check (gate-integrity)2. model-prompt3. codex-stage-result4. runtime-check (diff-integrity)5. runtime-check (server-typecheck)6. runtime-check (repository-lint)

One check before model-prompt, not two. Zero between model-prompt and codex-stage-result — they're adjacent.

Your falsifier is also not met. It required model-prompt, codex-stage-result, then the runtime-checks. There is a check at position 1, so the sequence isn't the coherent one you said would close the issue.

The underlying claim survives, and #117 should stay open. I nearly closed it on a plausible defense: gate-integrity sounds like a precondition, and a precondition legitimately preceding the prompt would make position 1 correct. repair kills that — the same openbot gate-integrity runs after codex-stage-result there, and in verify it heads each post-output check batch. The canonical pattern is model-prompt → codex-stage-result → [gate-integrity, diff-integrity, typecheck, lint, …]. Diagnose is the only stage that violates it.

Root cause, which is more specific than "ties":

stage	created_at	rows	ctid span
diagnose	21:09:55.847571	6	(14,6) → (27,3)
repair	21:10:38.167277	4	(15,6) → (28,3)
verify	21:10:50.525667	5	(29,1) → (29,5)
verify	21:11:02.399134	5	(29,6) → (30,3)

created_at DEFAULT now() is transaction_timestamp() — constant for every row in one transaction. So each insert batch ties completely, the sort degenerates to a no-op, and rows return in heap order. Heap order reflects free-space reuse: diagnose's gate-integrity landed in page 14 while its siblings went to pages 26–27, so it sorts to the front. Repair scattered the same way (model-prompt at (15,6)) but got lucky — the floated row belonged first anyway. Verify's batches are contiguous and render correctly.

That explains the 1-in-10 flake rate: which row lands in reclaimed space depends on heap state, which changes across restarts.

One constraint for the fix: insertion order isn't recoverable from this table. The only candidates are created_at (tied) and id (gen_random_uuid(), non-monotonic) — there's no serial or sequence. So sorting differently won't fix it. It needs either clock_timestamp() as the default, which differentiates rows within a transaction, or a monotonic sequence column with ORDER BY (created_at, seq). Adding a tiebreaker on existing columns would only make the wrong order deterministic.

Suggested issue text: not "checks render before the prompt" but "artifacts written in one transaction share transaction_timestamp(), so snapshot()'s ORDER BY created_at is a total tie and returns heap order; free-space reuse then surfaces arbitrary rows first — visible today as diagnose's gate-integrity preceding model-prompt."

On #111: yes, set up the MCP server with a slow tool. A side-effect probe shows the tool kept running; it doesn't show what the UI told the operator while it did, and on #108 the gap between "nothing visible happens" and an unhelpful red error turned out to be the reportable part. I didn't retest #108/#109 — say the word if you want me to check the rescoped #108 text reads right.

#111 is not browser-testable on this rig, and I should have checked sooner

I built a slow endpoint with a readable side effect (/slow?marker=X sleeps 8s then records; /markers lists) so the whole loop would be browser-observable with no terminal. Three routes to invoke it, all closed:

Chat UI + Stop button — the real test. Blocked: OPENAI_API_KEY and INTELLIGENCE_API_KEY are empty in .env, so no chat turn can run.
Register it as a custom MCP server — refused: "An MCP server must be reached over https."
open-web/get_json at localhost — refused: "That address is inside this deployment's own network."

I set AGENT_COMPUTER_ALLOW_PRIVATE_HOSTS=[REDACTED] expecting route 3 to open. It didn't, and I've removed it. That assumption was wrong and checking the code settled it: privateHostsAllowed appears zero times in builtin-web.ts, which reads process.env zero times at all. The refusal is unconditional.

That mismatch is worth something on its own — the README says the variable "lets a Bot reach this machine's own services," which reads deployment-wide but covers only the computer path. Filed as #118.

To unlock #111 in the browser I need one of: an OPENAI_API_KEY in .env (then the real Stop button works, which is the strongest version), or a publicly-reachable HTTPS MCP server. Your call — I'm not going to fake a third path.

Your point about why it matters stands and is why I won't settle for the script: a side-effect probe shows the tool kept running, but not what the operator was told while it did. On #108 that gap — "nothing visible happens" versus an unhelpful red error — was the reportable part.

What you can check in the browser right now

#117 — I was wrong, you were right, and the issue now carries your analysis. My prediction failed on both particulars (I said two checks before model-prompt and one between prompt and result; it's one and zero), and my falsifier wasn't met either. I rewrote the issue around your mechanism: created_at DEFAULT now() is transaction_timestamp(), the sort is a total tie, rows come back in heap order, and free-space reuse floats diagnose's gate-integrity into page 14 while its siblings sit in 26–27. I included your ctid table, the cross-stage evidence that kills the precondition defence (repair runs the same gate-integrity after codex-stage-result), and your constraint that insertion order is unrecoverable — so clock_timestamp() or a sequence column, not a tiebreaker. Title is now "Artifacts written in one transaction tie on created_at, so snapshot() returns heap order."

Worth re-reading to confirm I represented it correctly, since I got the first version wrong.

#118 — reproducible in one call. From DevTools on localhost:3030:

await (await fetch('/api/plugins/call', {method:'POST', headers:{'content-type':'application/json'}, body:JSON.stringify({ref:'open-web/get_json', agentId:'general-assistant', args:{url:'http://localhost:4313/markers'}})})).json()

I predict isError: true with the deployment's-own-network refusal. Falsifier: if it returns the JSON body, the refusal isn't unconditional and #118 is wrong. That endpoint is live on 4313 right now.

Unchanged: #108, #109, and the #32 behaviours you already tested. Yes — please do check the rescoped #108 text reads right; I rewrote it around your payload matrix and your bug-triage/visual-delivery probes, and you're better placed than I am to say whether it now describes what you actually saw.

Three tasks. Let me start with #118, including a control — if 4313 isn't actually reachable, an isError proves nothing about the refusal being unconditional.

Control passes: /markers returns 200 with {"count":0,"markers":[]}, and /slow hangs past 3s consistent with its 8s sleep. So the target is genuinely reachable. Now the plugin path:

Prediction confirmed. Let me also run it from the page context as specified, and verify the code claim underneath #118 — since the refusal being unconditional is the actual assertion, not just that it refused once.

Identical from the page context. Now verifying the unconditional part, which is the actual claim in #118:

The counts check out — but the refusal message isn't in that file. Absence of the flag there doesn't establish unconditionality if the guard lives elsewhere. Let me find where it actually is:

The refusal message lives in server/src/computer/target.ts:281, not builtin-web.ts — and privateHostsAllowed is defined and used, in config.ts:646 and :708. So grepping builtin-web.ts was looking in the wrong file. Let me read the actual guard:

Decisive: the refusal is not unconditional. Directly above it sits

The guard is gated on a caller-supplied option. So #118's stated mechanism is wrong. The real question is whether the plugin path propagates config into that option. Let me trace the callers:

The flag is plumbed widely, with routes.ts defaulting it to false. Now the key question — does the plugin path pass it at all?

Complete picture, and it corrects #118's mechanism. builtin-web.ts calls checkNavigationTarget(raw) at both line 113 and line 123 with no options argument. The parameter defaults to {}, so allowPrivateHosts is undefined → falsy → refusal.

So the flag isn't ignored globally — agents/endpoint.ts:42 passes it as true, and routes.ts/index.ts thread it from config. The open-web plugin path simply never consults it. Your empirical finding is correct and your conclusion holds, but the reason is a missing propagation on one path, not an unconditional guard. Now the issue text. Let me find the repo:

That's listing upstream. #117/#118 will be on the fork:

#117's body represents my analysis accurately. Let me see the remainder and the title:

#117 reads correctly, and it adds something I didn't: that factory_workflow_events carries the identical defect. Since that's attached to my analysis, let me verify it rather than let it ride:

Confirmed — both tables default to now() and both are sorted by asc(createdAt) (workflow-runtime.ts:494 and :499). The events extension is correct.

Now the rescoped #108:

#108 represents my measurements accurately, and it goes further — root-causing to the benchmark catalog, which I'd offered to chase but hadn't established. Let me verify that added claim:

Root cause verified: the catalog has exactly one kind: (ci-repair, entry ci-repair-v1), and factory_model_benchmarks holds only task=ci-repair. Let me spot-check the three line references, since a reader will jump to them:

model-router.ts:94-97 is accurate — it throws No benchmarked model clears the quality floor for ${input.task} when nothing is eligible. But the other two pointers are off. routes.ts:295 is the 400 validation branch — it's the exact error my hand-rolled curl hit last session, not where the 500 escapes. Let me find the real escape point:

routes.ts has no try/catch at all (only .catch(() => null) on JSON parsing). The uncaught escape is routes.ts:308 — await store.queueJob(...). And chooseModel is called at store.ts:156, not 150 (150 is the candidates query).

One more thing, because it may unblock #111. My #118 trace turned up agents/endpoint.ts:42 passing allowPrivateHosts: true — a path that deliberately permits private hosts:

namedAsAllowed permits private hosts when the host is explicitly named in an allowlist. That's a supported mechanism, not a bypass. Let me find the config key that feeds it:

All three checks done.

#118 — prediction confirmed, mechanism wrong.

The call refuses exactly as you predicted, isError: true with the deployment's-own-network text, identically via curl and from the page context. Control passed first: /markers returns 200 {"count":0,"markers":[]} from the shell and /slow hangs past 3s, so the target is genuinely live and the refusal isn't a dead endpoint.

But the refusal is not unconditional. computer/target.ts, immediately above the message:

Grepping builtin-web.ts looked in the wrong file — the guard isn't there, it's in computer/target.ts:281, and privateHostsAllowed is alive at config.ts:646/708. The actual defect is narrower and better: builtin-web.ts calls checkNavigationTarget(raw) at both line 113 and line 123 with no options argument, so the plugin path hardcodes the default false. The flag is honored elsewhere — agents/endpoint.ts:42 passes true, routes.ts/index.ts thread it from config. It's a dropped propagation on one path, not an absent feature. That also means the fix is small and local, which the current framing hides.

One more thing that weakens #118 as filed: README:396 already draws the exact distinction you say is missing — "list its address in AGENT_ENDPOINT_ALLOWED_HOSTS instead. That permits the one address, where the switch above permits the network." The docs complaint is thinner than stated; the code gap is the real finding. I'd retitle around open-web dropping allowPrivateHosts.

#117 — represented correctly. The ctid table, the cross-stage evidence, the precondition rebuttal, and the unrecoverable-insertion-order constraint are all faithful, and the acceptance criteria correctly rule out a uuid tiebreaker. I verified your addition, since it's attached to my analysis: both tables default created_at to now(), and both are sorted asc(createdAt) at workflow-runtime.ts:494 and :499. The events extension holds.

#108 — accurate, root cause verified, two line refs off. The payload matrix, the text-destructive quote, and the "feedback exists but carries no information" reading are all faithful. Your added root cause checks out: the catalog has exactly one kind: (ci-repair, entry ci-repair-v1), and factory_model_benchmarks holds only task=ci-repair. Corrections:

model-router.ts:95 — right, throws when eligible.length === 0.
store.ts:150 → 156. 150 is the candidates query; chooseModel is called at 156.
routes.ts:295 → 308. 295 is the 400 validation branch — the exact error my malformed curl hit last session. The uncaught escape is await store.queueJob(...) at 308; routes.ts has no try/catch at all, only .catch(() => null) on JSON parsing.

Also the benchmark count now reads 7, not the 4 in the issue — drift from the ci-repair run my Test 3 created. Worth a parenthetical so nobody thinks the number is wrong.

#111 — I'd rather leave it blocked than weaken it. I won't supply an API key, and I'd push back gently on the framing that route 3 was ever close to sufficient: reaching /slow through open-web would only re-demonstrate what your side-effect probe already showed. The thing worth measuring is what the operator sees while the tool keeps running, and only the chat UI with a real Stop button produces that. So the key is not one of two options, it's the only one — your call whether to add it.

For completeness: AGENT_ENDPOINT_ALLOWED_HOSTS=localhost:4313 is the documented way to reach a private host (it flows into namedAsAllowed, which passes allowPrivateHosts: true behind an exact-match allowlist). That's the supported mechanism you were reaching for with the wrong variable. It does not unblock #111 — a chat turn still needs a key — so I'd not spend time on it except as evidence for #118.

Everything is now checkable in a browser. Here's what changed, and the one thing that still isn't.

New: the mutation check is no longer something you take on trust

My criterion-2 mutation check existed only as terminal output I pasted. That's exactly the "trust me" problem you've been pushing on. So I converted it into a CI result you can read.

PR #119 — draft, titled "DO NOT MERGE", deliberately broken. It applies the mutation #32 criterion 2 specifies — the refund moved out of the transaction, not deleted — and nothing else. One file, 15 insertions, 2 deletions.

CI is red, and the log says why:

expect([...live.seen]).not.toContain("pending:1"); error: expect(received).not.toContain(expected) Expected to not contain: "pending:1" Received: [ "running:1", "pending:1" ] (fail) a concurrent reader never sees a reset stage still holding the attempt it was given back

tests job log (https://github.com/gagan114662/openbot/actions/runs/33926323114/job/101195523665)

An assertion failure naming the exact state the issue describes — not a compile error, which is what criterion 2 explicitly rules out. PR #116 is green on the same tests, so the pair is the whole proof: green when the product is correct, red on this specific defect. Neither half requires you to run anything.

If it had come back green, the test in #116 wouldn't catch what it claims and criterion 2 would be unproven. Close #119 once you've looked.

Read: three issue bodies I changed since your last run

Nothing in the running app changed, so there's nothing to re-drive in the UI. What changed is text, and the thing worth checking is whether it represents your measurements honestly:

#118 — retitled and rebuilt around your correction. The claim is now the dropped propagation at builtin-web.ts:113 and :123, not an unconditional guard. I removed the documentation complaint since README:396 already draws the distinction.
#108 — store.ts:150→156, routes.ts:295→308, plus grep -c 'try {' returning 0. The benchmark count is now phrased as an invariant ("every row is task = ci-repair") rather than a number, since it drifts.
#117 — unchanged; you already verified it.

Still not browser-testable

#111. Blocked on an OPENAI_API_KEY, and I've stopped pretending there's a second path. Your call.

The criterion-3 mutation check (disabling the attempt-ceiling in claim) is still only in my paste. I can give it the same treatment — a second red draft PR — but I didn't want to put two deliberately-broken PRs in your list without asking. Say the word.

Rig, if you want the earlier tests again

Server :3021, app :3030, slow endpoint :4313, admin page returning 200. The #118 repro from DevTools still works and I re-confirmed the endpoint is live.

Let me verify PR #119 — the diff being minimal and the failure being an assertion (not a compile error) is the whole claim.

That doesn't match the description: you said one file, 15 insertions, 2 deletions. It's 2 files, 319 additions, 4 deletions — and one of them is the test file. For a mutation check that matters: if the test changed too, a red result could come from the test edit rather than the product mutation. Let me read the diff.

The product mutation is exactly right: the refund is lifted out of the transaction and moved to a statement after it commits. That's criterion 2's mutation precisely. But I need to know whether the failing test already exists in #116, or is introduced by #119 — that decides whether this proves #116's test catches the defect, or merely that a test does.

Not on main — and added in #119. That's expected only if #116 (unmerged) is what introduces it. Let me check whether #116 contains the identical test:

Structure is sound: #116 adds the test only (1 file, 304/2, green); #119 adds the same test plus the mutation (2 files, 319/4). The delta is exactly 15 additions, 2 deletions — so your "15 insertions, 2 deletions" describes the product file correctly; it was the "one file" PR summary that was off, since #119 must carry #116's test to run it.

The remaining risk is that #119's copy of the test differs from #116's. Let me check:

That "IDENTICAL" is bogus — zsh glob-expanded the ?, both files came back empty, and I diffed two empty files. Redoing with quoting:

Genuinely identical now — both 64194 bytes, byte-for-byte. So #119 is exactly #116's test plus the product mutation. The comparison is clean. Now the CI results:

#119: tests fail, verify fail, but build passes and format, lint, types passes — which is the load-bearing detail, since it rules out a compile error. #116: everything green including tests. Let me confirm the failure text and check what verify is:

The quoted failure is exactly as described — an assertion at line 2031, not a compile error:

2031 |  expect([...live.seen]).not.toContain("pending:1");Expected to not contain: "pending:1"Received: [ "running:1", "pending:1" ](fail) a concurrent reader never sees a reset stage still holding the attempt it was given back

2031 |  expect([...live.seen]).not.toContain("pending:1");Expected to not contain: "pending:1"Received: [ "running:1", "pending:1" ](fail) a concurrent reader never sees a reset stage still holding the attempt it was given back

But there's a second failure you didn't mention — two worker processes race and only the owning session can commit or fail the stage, failing on toMatchObject. Let me get the full failure count and check the verify job:

Full picture: 2474 pass, 2 fail. Both failures are caused by the mutation (the identical test file is green on #116), and verify is just an aggregate gate — TEST_RESULT: failure, everything else success. So it's not an independent defect.

The proof holds. One amendment: two tests fail, not one. Now the revised issue bodies:

#118 represents my correction faithfully. It adds one citation I didn't make — connection-test.ts:101,111 — so let me verify it:

connection-test.ts:100-101,111-113 does thread the flag — citation accurate. And #108's corrections are all applied, including the explicit notes that :150 is the candidates query and :295 is the 400 branch, the no try { finding, and the count reworded as an invariant.

Closing #119 as you asked: