Ideated blog series on agent-friendly CLI
Public
Ideated blog series on agent-friendly CLI
Agent
Pi
Model
GPT-5.5
Started
Aug 24, 7:35 AM
Messages32
Downloads0
Workspace
Outlined blog series themes and narrative arc.
PlanDiscussed experimental design and overfitting safeguards.
UnderstandSynthesized core principles from experiment results.
BuildYou are a fresh Pi session for BLOG IDEATION, not implementation.
Goal: help Priyanga shape a blog post about an experiment in /Users/priyangapkini/nilenso/ai-playground/autoresearch answering: “what makes a CLI agent-friendly?”
Do not run paid experiments. Do not modify code unless asked. Read files as needed, especially:
Essential experiment context:
botmap commands through a shell.autoresearch/agenteval/ with record-v2 artifacts, taxonomy, probes, sabotage fixtures, scoring, explanations, enrichment, analysis."class": null.experiments/runs/agenteval-measurement-3009509/experiments/runs/agenteval-measurement-3009509-retry-incomplete/; 9 retries, 2 completed, cost ~$0.31.Cambridge, MA and Malta, MT can be US state abbreviations as well as ISO country codes; evaluator bugs can manufacture “findings.”00bff1a Warn when categories output is truncated.experiments/paired/categories-truncation-hint/result.md.B. Count wrong-column hint
7c794ff Hint when count filter uses class and subtype wrong.count returns zero for class=X or subtype=X, it checks swapped field and emits a concrete correction.0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844. Try --where class=bicycle_parking...experiments/paired/count-wrong-column-hint/result.md.botmap/data/skill.md only, using new evaluator.--all-files now excludes evals/*, tests/eval_fixtures/*, tests/test_eval_*.Your task:
Start by reading the key docs and then give a concise ideation memo.
Planning file reading approach
Planning concise memo structure
Agent-friendly CLI design is recovery design
The worst CLI failure is not a crash — it is a polite wrong zero
Before optimizing agents, build an evaluator that can see silent wrongness
Warnings on stderr are an agent protocol
Agent-friendly CLIs teach their ontology
Agent-friendly CLI design is recovery design.
This is the broadest and most defensible. It lets you include the evaluator story, the paired experiments, the stderr/stdout design insight, and the uncomfortable evaluator-bug correction without making overbroad claims.
Opening contrast:
We usually design CLIs for humans: terse output, compact help text, trust that the user knows when to retry. But agents use CLIs differently. They compose commands from partial understanding, inspect outputs literally, and often treat silence as truth.
Then introduce the motivating question:
What makes a CLI agent-friendly?
Clarify that this was studied through botmap, where agents receive plain-English map questions and must discover/compose CLI commands.
Core point:
Good line:
A CLI that crashes is rude. A CLI that returns
0for the wrong query is dangerous.
Use the “polite wrong zero” motif here.
Introduce autoresearch/agenteval/:
Explain the important classes:
| Class | Meaning | Blog framing |
|---|---|---|
| A | hard unguided failure | CLI refuses without useful recovery |
| B | guided recoverable failure | CLI says what to do next |
| C | silent wrong | CLI succeeds but misleads |
| D | degenerate route | answer possible but inefficient/unusable |
| E | environment/quota | excluded from CLI blame |
| F | agent-side failure | recorded, not charged to CLI |
Key idea:
The taxonomy was not just a scorer. It became a generator of design requirements.
Use the corrected combined histogram:
{ "records": 60, "attempts_with_failures": 34, "class_counts": {"clean": 389, "C": 55, "B": 15, "A": 38, "D": 3}, "subtype_counts": { "c-truncated": 25, "c-unknown": 25, "c-wrong-type": 3, "c-wrong-column": 2 }}Useful framing:
C silent wrong appeared 55 timesc-truncatedc-unknown remains an instrumentation work queue, not a product conclusionPossible figure: Stacked bar chart of call classes: clean, A, B, C, D.
Possible table: C subtype counts before any paired fixes.
Property:
An agent-friendly CLI says when output is truncated and names the recovery action.
Evidence from experiments/paired/categories-truncation-hint/result.md:
Before:
{"c-truncated": 13, "B": 5, "A": 3, "c-wrong-column": 1, "c-unknown": 7}After:
{"c-truncated": 5, "B": 2, "c-unknown": 2, "A": 1}Change:
[botmap] Showing top N of TOTAL categories. This list is truncated; rerun with`--top TOTAL` or a larger --top before concluding a category is absent.Claim:
c-truncated fell 13 → 5 on matched subset.Design insight:
stderr can carry recovery guidance while stdout JSON remains machine-stable.
Property:
If a filter value exists in another field, the CLI should name that field and give the corrected filter.
Evidence from experiments/paired/count-wrong-column-hint/result.md:
Before:
{"c-truncated": 5, "B": 3, "c-wrong-column": 2, "c-unknown": 6, "A": 1, "c-wrong-type": 1}After:
{"c-truncated": 3, "B": 2, "c-vocabulary": 2, "A": 1}Concrete hint:
[botmap] 0 rows for subtype='bicycle_parking', but class='bicycle_parking'returns 1,844. Try `--where class=bicycle_parking` before concluding none exist.Claim:
c-wrong-column fell 2 → 0 on matched subset.Design insight:
Agents need CLIs to teach them the schema while they use it.
Include the MA/MT false positive story.
The evaluator initially classified some entity resolutions as c-wrong-entity because it treated:
Cambridge, MAMalta, MTas ISO country-code-like ambiguity before recognizing them as valid US state abbreviations.
Correction:
89acc62Why this matters:
If your evaluator is supposed to reveal product flaws, evaluator bugs become fake product insights.
This is a strong credibility section. It shows the post is not just victory-lap engineering.
Mention current arms briefly:
Important lesson:
Full-repo optimization is powerful, but it can accidentally optimize the yardstick. The evaluator must be read-only to the optimizer.
This ties the experiment to broader AI-evaluation practice.
Possible closing list:
An agent-friendly CLI should:
Closing line:
The agent-friendly CLI is not the one with the most features. It is the one that makes wrong turns observable and recoverable.
Baseline measurement table
docs/agent-friendly-cli.md, experiments/orchestrator/HANDOVER.mdFailure taxonomy table
Before/after truncation chart
c-truncated: 13 → 5experiments/paired/categories-truncation-hint/result.mdBefore/after wrong-column chart
c-wrong-column: 2 → 0experiments/paired/count-wrong-column-hint/result.mdExample stderr hints
Evaluator architecture diagram
“Property from failure” table
c-truncated → never silently truncatec-wrong-column → if value exists elsewhere, say wherec-unknown → instrumentation gapc-truncated was a major observed failure subtype in the baseline.c-truncated from 13 to 5 on the paired subset.c-wrong-column from 2 to 0 on the paired subset.I want to create a series of blog post 1. understanding GEPA - optimise anything 2. Choosing the evalutor - scoring/weights 3. What makes a CLI agent-friendly? wdyt
Starting response ideation
I think that sequence is strong, but I’d slightly reframe it so the series has a clear narrative progression:
We wanted to optimize a CLI for agents. Then we realized the optimizer is only as good as the evaluator. Then the evaluator revealed what “agent-friendly” actually means.
Role in series: introduce the optimization loop.
Possible thesis:
GEPA is interesting because it treats improvement as search over arbitrary artifacts — prompts, skill files, code, docs — as long as you can define an evaluator.
What to cover:
Ending hook:
Before asking GEPA to improve the CLI, we had to ask a harder question: improve according to what?
Role in series: explain why evaluation design became the real work.
Possible thesis:
The evaluator is not a scoreboard; it is the definition of what you care about.
This should probably be the most technical / methodological post.
What to cover:
agenteval system:
Ending hook:
Once the evaluator could see silent wrongness, it started producing design requirements. That led to the real question: what makes the CLI itself agent-friendly?
Role in series: turn experiment results into product/design principles.
Possible thesis:
Agent-friendly CLI design is recovery design.
This post should be more accessible and design-oriented.
What to cover:
c-truncated 13 → 5c-wrong-column 2 → 0Ending:
A CLI is not agent-friendly because it has more commands. It is agent-friendly when it makes wrong turns observable and recoverable.
Something like:
Or:
Your proposed three posts are exactly the right three topics. I’d just make sure Post 1 does not overclaim GEPA as the hero.
The real narrative should be:
That makes the series much stronger than “we ran an optimizer and found tips.” It becomes a story about experimental discipline.
How we are caustioning against overfitting
We’re cautioning against overfitting in a few concrete ways. This could become a recurring thread across all three posts.
Most important guardrail:
The optimizer must not be allowed to edit the exam.
For Arm C, broad repo context is allowed, but evaluator files are excluded from the editable set:
evals/*tests/eval_fixtures/*tests/test_eval_*So the optimizer can inspect enough context to make useful changes, but it cannot improve its score by changing the yardstick.
The new evaluator has sabotage fixtures: known-bad cases that must be caught.
Purpose:
If an invisible failure classifies as clean, the evaluator fails before optimization begins.
This guards against evaluator drift and “score looks good because detector stopped detecting.”
Existing runs are not rewritten or rescored in place.
Instead:
record.json stays untouchedrecord-v2.jsonThis preserves auditability and avoids retroactively making the experiment look cleaner than it was.
Claims are not based on vibes or one-off anecdotes.
For a property to move from hypothesis to confirmed, we require:
Example:
c-truncated 13 → 5c-wrong-column 2 → 0This reduces the chance of mistaking a lucky run for a design principle.
The docs explicitly separate:
That prevents overclaiming from cheap probe evidence.
Good phrasing:
A probe gives us a hypothesis. A paired run gives us a claim.
Classes E and F prevent charging the CLI for things it did not cause:
This avoids optimizing the CLI to fix noise, quota failures, or agent stubbornness.
c-unknown instead of forcing explanationsc-unknown is important.
Instead of pretending every empty result is understood, the evaluator says:
We saw a suspicious empty result, but our probes cannot yet explain it.
That prevents fake certainty and keeps instrumentation gaps visible.
The MA/MT false positive is a strong anti-overfitting story.
The evaluator initially treated Cambridge, MA and Malta, MT incorrectly, manufacturing a c-wrong-entity finding.
Correction:
This shows the experiment includes correction loops, not just score chasing.
The plan explicitly delayed scoring weights until after observing real class distribution.
That avoids designing weights around imagined failures or desired outcomes.
The paired results are deliberately worded as:
That’s a communication guardrail against overclaiming.
A concise line for the blog:
We tried to prevent overfitting at three levels: don’t let the optimizer edit the evaluator, don’t trust the evaluator until sabotage cases pass, and don’t call a property confirmed until it survives a paired before/after run.
Are we keeping a heldout set for testing? Or are we running the same experiment on same set of questions?
Proposing train/dev/test split
From the docs/context you gave: we are mostly running paired experiments on the same question set/subsets, not yet maintaining a clean held-out test set.
That is okay for the current phase, but it means we should be careful about what we claim.
We ran the baseline on the fixed question bank:
This gives the failure distribution.
We are intentionally using the same questions before and after a candidate change.
Example:
bike-parking-coverage, basic-category-rollup, bus-stops-cambridgebike-parking-coverage, residential-share-cambridgeThat is the right design for asking:
Did this change reduce this observed failure mode under matched conditions?
But it is not the same as proving generalization to unseen questions.
We currently have:
At this stage we were not trying to prove broad generalization. We were doing paired experiments on known failure cases: same question, same conditions, one change. That lets us test whether a proposed CLI property fixes the failure it was designed to fix. A held-out set is still needed before claiming the property generalizes across the whole task distribution.
For the series, I’d explicitly introduce three sets:
The current 30-question bank / Phase 4 run.
Purpose:
Small matched subsets from the discovery set.
Purpose:
This is what we have already done.
Not yet clearly present, but we should add it before stronger claims.
Purpose:
Current safe claim:
On matched subsets, these changes reduced specific evaluator-detected failure classes.
Not yet safe:
These changes improve agent performance generally across map questions.
Future held-out claim, if added:
The properties discovered on the development set also reduced silent wrongness on unseen questions.
SYNTHESIS REQUEST: answer the core research question.
Stop any experimental execution. Do not run paid work. Do not run GEPA. Do not run autoresearch.runner. Do not modify implementation code.
Question to answer: What are the general principles from the experiments so far that help answer: what makes a CLI agent-friendly?
Use evidence from the autoresearch/botmap experiment, especially:
Your answer should separate:
For each principle include:
Write your answer to one file:
Create the directory if needed. After writing, reply with the file path and top 5 principles only.
Preparing blog ideation environment
Creating structured Markdown document
Wrote: experiments/blog-ideation/GENERAL-PRINCIPLES.md
Top 5 principles:
{ "records": 60, "attempts_with_failures": 34, "class_counts": {"clean": 389, "C": 55, "B": 15, "A": 38, "D": 3}, "subtype_counts": { "c-truncated": 25, "c-unknown": 25, "c-wrong-type": 3, "c-wrong-column": 2 }}{"c-truncated": 13, "B": 5, "A": 3, "c-wrong-column": 1, "c-unknown": 7}{"c-truncated": 5, "B": 2, "c-unknown": 2, "A": 1}[botmap] Showing top N of TOTAL categories. This list is truncated; rerun with`--top TOTAL` or a larger --top before concluding a category is absent.{"c-truncated": 5, "B": 3, "c-wrong-column": 2, "c-unknown": 6, "A": 1, "c-wrong-type": 1}{"c-truncated": 3, "B": 2, "c-vocabulary": 2, "A": 1}[botmap] 0 rows for subtype='bicycle_parking', but class='bicycle_parking'returns 1,844. Try `--where class=bicycle_parking` before concluding none exist.