Read and analyzed the article's gaps.
UnderstandDistinguished operational gaps from open problems.
UnderstandChecked current evidence and industry status.
Verifyhttps://murraycole.com/posts/software-factory how far are we from adressing all the gaps mentioned here?
I’ll read the article, identify its gaps, and check current evidence for how much progress has been made. I’m using the agent-reach skill for the research.
The article mixes gaps we can address with engineering today—environments, checks, and failure reporting—with harder questions about whether those checks capture everything that matters. I’m checking that distinction against production reports. Agent-reach’s search backend isn’t configured here, so I’m using web search.
We can address most of the operational gaps today. Reliably closing all of them—enough to leave arbitrary production software entirely unattended—is still an open problem with no defensible completion date.
Assuming “we” means the industry, that’s my assessment as of September 12, 2026. The article was updated September 5, so its concerns are current.
The important distinction is between having a mitigation and having enough evidence to remove human oversight.
| Gap in the article | How close are we? | What remains |
|---|---|---|
| Reproducible environments and isolated execution | Implementable today. | Making each company’s setup, dependencies, test data, and integrations reproducible is still substantial work. |
| Secrets and prompt injection | Containment is available; prevention remains incomplete. | Keeping credentials outside the sandbox helps, but an agent can still misuse an authorized tool. Permissions must constrain actions as well as credential access. |
| Back-pressure: lint, types, tests, policy checks | Implementable today. | Checks must run independently and block release; asking the agent to run them is insufficient. |
| Silent failures and replayability | Largely an engineering problem today. | External status tracking, timeouts, recorded checks, and preserved artifacts help. Reconstructing a run does not guarantee identical model behavior on replay. |
| Spec quality and missing context | Partially addressable. | Agents can help refine requirements, but someone must decide intended behavior, acceptable tradeoffs, and whether the requested feature is useful. |
| Agents grading their own work | Mitigations exist, with residual uncertainty. | Protected tests, independent evaluation, and human calibration help. A second model can share the first model’s blind spots. |
| Generation outrunning verification | Still the central bottleneck. | Generating more tests is easier than establishing that they cover the failures customers will encounter. |
| Code erosion and maintenance cost | Detectable in part; not solved. | Passing today’s tests says little about the cost of the next ten changes. Architecture and evolving requirements remain difficult. |
| Velocity theater and economics | Measurable today, incompletely demonstrated industry-wide. | Track accepted outcomes, defects, rework, review time, and total cost—not PR counts. |
| Progressive autonomy and adoption | Available for bounded work. | Expanding autonomy requires evidence per task category and risk level. Non-engineer access doesn’t eliminate engineering ownership. |
Three pieces of evidence explain why I draw that line:
I’d also qualify two claims in the article. Using a different reviewer model does not guarantee independent judgment. And hiding secrets does not automatically contain every attack. Anthropic’s September incident assessment explicitly says evaluations that reliably expose every deployment failure remain unsolved. Those incidents occurred in cybersecurity evaluations, so they shouldn’t be treated as ordinary coding failure rates. September assessment
For planning, my estimates—not measured industry timelines—would be:
The milestone I’d use is “this class of changes ships with acceptable defects, rework, and cost over repeated releases.” That’s a stronger basis for increasing autonomy than either model benchmark scores or the number of generated PRs.