Summary

Closed, Completed

Understand
Reviewed key documentation and experiment results.
Understand
Build
Synthesized general principles for agent-friendly CLI.
Build
Edit
Documented top principles in a summary file.
Edit

You are a fresh Pi session named DERIVATION. Your job is not implementation and not blogging style. Your job is to derive principles for making an agent-friendly CLI from the autoresearch/botmap experiments so far.

Work in repo: /Users/priyangapkini/nilenso/ai-playground/autoresearch

Do not run paid experiments. Do not modify code unless explicitly asked. Read files as needed.

Key files to read first:

docs/plan.md
docs/agent-friendly-cli.md
experiments/orchestrator/HANDOVER.md
experiments/paired/categories-truncation-hint/result.md
experiments/paired/count-wrong-column-hint/result.md
autoresearch/agenteval/contract.py
autoresearch/agenteval/taxonomy.py
autoresearch/agenteval/score.py
autoresearch/agenteval/explain.py
autoresearch/agenteval/probe.py

Experiment goal: Derive what makes a CLI friendly to AI agents. The concrete testbed is botmap, a CLI over Overture map data. Agents get plain-English map questions and must discover/compose the right commands via shell use. The output we want is principles grounded in measured traces, not vibes.

Important constraints/context:

Do not touch botmap/evals/.
New evaluator lives in autoresearch/agenteval/ and is now the intended judge.
Clean record-v2 calls use JSON "class": null.
Quota/environment failures are attempt-level verdicts and excluded from CLI blame.
Class F is agent-side and recorded separately, not charged to CLI.
Global Claude botmap skill remains disabled because it shadowed project-scoped skill files.

Evaluator/taxonomy summary:

A = hard/unguided failure, e.g. command exits with usage/raw traceback/no useful recovery.
B = guided/recoverable failure: CLI gave a usable hint/recovery path.
C = silent wrong: command exits successfully or plausibly but leads the agent to a wrong answer. Subtypes include c-truncated, c-wrong-column, c-wrong-type, c-wrong-entity, c-vocabulary/c-unknown.
D = degenerate route, e.g. unnecessarily broad/bulk route.
E = environment/quota; excluded from CLI blame.
F = agent-side; recorded separately, not charged to CLI.
Scoring rubric: correctness/recoverability 60%, token efficiency 20%, wall-clock 20%; self-recovery is first-class.
Sabotage fixtures exist so evaluator must catch known invisible failures before optimization.

Baseline/BEFORE measurement:

Run: experiments/runs/agenteval-measurement-3009509/
Retried incomplete attempts separately: experiments/runs/agenteval-measurement-3009509-retry-incomplete/
Combined corrected summary: experiments/runs/agenteval-measurement-3009509/agenteval-summary-with-retries.json
60 records, 34 attempts with failures.
Class counts after correction: clean 389, C 55, B 15, A 38, D 3.
C subtype counts: c-truncated 25, c-unknown 25, c-wrong-type 3, c-wrong-column 2.
Important evaluator correction: initial c-wrong-entity counts were polluted by MA/MT ambiguity. Cambridge, MA and Malta, MT can be US state abbreviations, not ISO country codes. False evaluator findings are a first-class risk.

Paired experiments completed:

Categories truncation hint

Artifact: experiments/paired/categories-truncation-hint/result.md
Botmap candidate: 00bff1a Warn when categories output is truncated
Change: when botmap categories output is capped, emit stderr hint while preserving JSON stdout.
Paired result: target c-truncated failures dropped from 13 BEFORE to 5 AFTER.
Derived candidate principle: never silently truncate output; if output is capped, say so machine-readably/actionably and tell the agent how to get complete data.
Status: provisionally confirmed; not a full task-success guarantee because other failures remain.

Count wrong-column hint

Artifact: experiments/paired/count-wrong-column-hint/result.md
Botmap candidate: 7c794ff Hint when count filter uses class and subtype wrong
Change: if count returns zero for class=X or subtype=X, test the swapped field and emit correction on stderr.
Example: 0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844. Try --where class=bicycle_parking before concluding none exist.
Paired result: c-wrong-column fell from 2 BEFORE to 0 AFTER on matched subset.
Derived candidate principle: if a value exists elsewhere, say where; zero is not enough when the CLI can detect likely field confusion.
Status: confirmed narrowly.

Current arms/runs, for context only:

Arm A: loop-as-a-skill / existing candidate screening using new evaluator; current run experiments/runs/arm-a-new-evaluator-count-zero-hint-9a2496d/.
Arm B: prompt-lever GEPA, edits only botmap/data/skill.md; run experiments/runs/prompt-3009509-1787544952/.
Arm C: full repo context + broad edit permissions but evaluator files excluded; latest valid run experiments/runs/tool-3009509-1787550419/.
Arm C correction: optimizer must not be allowed to propose changes to evaluator/yardstick. Full repo context is okay read-only; editable set excludes evals/*, tests/eval_fixtures/*, tests/test_eval_*.

Your derivation task:

Read the key evidence files.
Derive a principled framework for “agent-friendly CLI” from the traces and paired experiments.
Separate:
- confirmed principles
- plausible but not yet proven principles
- evaluator/instrumentation principles
- anti-principles / traps
For each principle, provide:
- name
- short statement
- failure class/subtype it addresses
- evidence from current experiment
- design implication for CLI authors
- what paired experiment would strengthen/weaken it
Pay special attention to the difference between human-friendly and agent-friendly CLI design.
Do not overclaim. If evidence is subset-based, say so.
Produce a concise derivation memo and a list of next experiments that would make the principles stronger.

Start by reading docs/agent-friendly-cli.md and the two paired result files, then produce your derivation memo.

>python3 - <<'PY'
import json, collections
p='experiments/runs/agenteval-measurement-3009509/agenteval-summary-with-retries.json'
d=json.load(open(p))
by=collections.defaultdict(list)
for detail in d['details']:
  for f in detail['failures']:
    key=f.get('subtype') or f.get('class')
    if len(by[key])<3:
      by[key].append((detail['attempt'], f.get('index'), f.get('argv'), f.get('evidence')))
for k, vals in by.items():
  print('\n',k)
  for v in vals:
    print(v)
PY

c-truncated
('asian-restaurants-rollup__r1', 1, ['--json', 'categories', '-t', 'place', '--in', 'Cambridge, MA', '--top', '100'], 'c-truncated: original output length equalled limit 100; raised limit returned 980 rows')
('asian-restaurants-rollup__r1', 2, ['--json', 'categories', '-t', 'place', '--in', 'Cambridge, MA', '--top', '400'], 'c-truncated: original output length equalled limit 400; raised limit returned 980 rows')
('asian-restaurants-rollup__r2', 1, ['--json', 'categories', '-t', 'place', '--in', 'Cambridge, MA', '--top', '200'], 'c-truncated: original output length equalled limit 200; raised limit returned 980 rows')

 c-unknown
('asian-restaurants-rollup__r1', 15, ['--json', 'count', '-t', 'place', '--in', 'Cambridge, MA', '--where', 'categories.primary=indonesian_restaurant'], 'empty result with no conclusive probe; reason unknown')
('asian-restaurants-rollup__r1', 17, ['--json', 'count', '-t', 'place', '--in', 'Cambridge, MA', '--where', 'categories.primary=mongolian_restaurant'], 'empty result with no conclusive probe; reason unknown')
('asian-restaurants-rollup__r1', 19, ['--json', 'count', '-t', 'place', '--in', 'Cambridge, MA', '--where', 'categories.primary=himalayan_restaurant'], 'empty result with no conclusive probe; reason unknown')

 B
('basic-category-rollup__r2', 6, ['--json', 'capabilities'], 'tool named a recovery path for the next iteration')
('beach-accessibility-malta__r1', 14, ['--json', 'capabilities'], 'tool named a recovery path for the next iteration')
('beach-accessibility-malta__r1', 15, ['--json', 'capabilities'], 'tool named a recovery path for the next iteration')

 A
('basic-category-rollup__r2', 8, ['where', 'Cambridge, MA', '--geometry'], 'command failed without recovery guidance')
('beach-accessibility-malta__r1', 6, ['--json', 'count', '-t', 'land_use', '--in', 'Malta, MLT', '--where', 'class=beach'], 'Usage: python -m botmap count [OPTIONS]')
('beach-accessibility-malta__r1', 7, ['--json', 'count', '-t', 'land_use', '--in', 'Malta, MLT', '--where', 'subtype=beach'], 'Usage: python -m botmap count [OPTIONS]')

 c-wrong-type
('beach-accessibility-malta__r1', 8, ['--json', 'count', '-t', 'land_use', '--bbox', '13.936042785644531,35.585365295410156,14.822786331176758,36.28254699707031', '--where', 'class=beach'], 'c-wrong-type: land_use returned 0; land returned 65 for the same filter')
('beach-accessibility-malta__r1', 11, ['--json', 'count', '-t', 'water', '--bbox', '13.936042785644531,35.585365295410156,14.822786331176758,36.28254699707031', '--where', 'class=beach'], 'c-wrong-type: water returned 0; land returned 65 for the same filter')
('residential-share-cambridge__r2', 19, ['--json', 'count', '-t', 'building', '--in', 'Cambridge, MA', '--where', 'subtype=military'], 'c-wrong-type: building returned 0; land_use returned 1 for the same filter')

 c-wrong-column
('bike-parking-coverage__r1', 8, ['--json', 'count', '-t', 'infrastructure', '--in', 'Cambridge, MA', '--where', 'subtype=bicycle_parking'], 'c-wrong-column: subtype=bicycle_parking returned 0; class=bicycle_parking returned 1844. The value exists in a different column.')
('residential-share-cambridge__r2', 12, ['--json', 'count', '-t', 'building', '--in', 'Cambridge, MA', '--where', 'subtype=government'], 'c-wrong-column: subtype=government returned 0; class=government returned 23. The value exists in a different column.')

 D
('ev-charging-gap__r1', 6, ['where', 'Cambridge, MA', '--geometry'], 'command eventually answered but by an unusable route')
('hardware-near-bikepaths__r2', 7, ['where', 'Cambridge, MA', '--geometry'], 'command eventually answered but by an unusable route')
('tall-buildings-cambridge__r2', 3, ['--json', 'containing', '42.3653,-71.0649'], 'command eventually answered but by an unusable route')

DERIVATION memo: agent-friendly CLI principles from autoresearch/botmap

Core thesis

An agent-friendly CLI is not mainly “feature-rich” or “pleasant for humans.” It is observable, self-describing, and recovery-oriented under shell composition. The botmap traces show agents often fail not because the command is impossible, but because the CLI returns an ambiguous success, a silent zero, a capped list, or a generic error that gives no next move.

For humans, terse output is often acceptable because humans bring domain knowledge and can notice suspicious absence. For agents, absence is data. A successful empty result or capped list is easily incorporated into a confident wrong answer unless the CLI makes uncertainty explicit and tells the agent how to verify or recover.

Evidence base read:

docs/agent-friendly-cli.md
docs/plan.md
experiments/orchestrator/HANDOVER.md
paired results:
- experiments/paired/categories-truncation-hint/result.md
- experiments/paired/count-wrong-column-hint/result.md
evaluator files:
- autoresearch/agenteval/contract.py
- taxonomy.py
- score.py
- explain.py
- probe.py
combined baseline:
- experiments/runs/agenteval-measurement-3009509/agenteval-summary-with-retries.json

Baseline corrected distribution:

60 records, 34 attempts with failures
class counts: clean 389, C 55, B 15, A 38, D 3
C subtypes: c-truncated 25, c-unknown 25, c-wrong-type 3, c-wrong-column 2

Confirmed principles

1. Never silently truncate output

Statement: If output is capped, the CLI must say it is capped and name the exact way to get complete output.

Addresses: Class C / c-truncated.

Evidence:
Baseline showed many category discovery calls where output length equalled --top, and a limit-raise probe found more rows. Example: categories -t place --in Cambridge, MA --top 100 returned 100 rows; raised limit returned 980.

Paired experiment:

subset: bike-parking-coverage, basic-category-rollup, bus-stops-cambridge, 2 repeats each
change: botmap emitted stderr hint:

text

[botmap] Showing top N of TOTAL categories. This list is truncated; rerun with`--top TOTAL` or a larger --top before concluding a category is absent.

[botmap] Showing top N of TOTAL categories. This list is truncated; rerun with`--top TOTAL` or a larger --top before concluding a category is absent.

result: c-truncated fell from 13 BEFORE to 5 AFTER.

Design implication:
Any CLI producing lists must expose:

whether the list is complete,
total available count when known,
exact pagination/limit/complete-data command,
ideally in structured output or stable stderr without corrupting JSON stdout.

Strengthening experiment:
Run the truncation hint across the full question bank and measure not only subtype reduction but final-answer correctness, recovery commands, tokens, and wall-clock.

Weakening experiment:
If full-bank paired runs show agents still make absence conclusions despite the hint, the principle may need refinement: hints must be machine-readable or included in JSON metadata, not just stderr prose.

2. If a value exists elsewhere, say where

Statement: When a zero result uses a real value in the wrong field, the CLI should name the field where the value exists and provide the corrected filter.

Addresses: Class C / c-wrong-column.

Evidence:
Baseline examples:

subtype=bicycle_parking returned 0, but class=bicycle_parking returned 1,844.
subtype=government returned 0, but class=government returned 23.

Paired experiment:

subset: bike-parking-coverage, residential-share-cambridge, 2 repeats each
change: botmap count tested swapped class/subtype field on zero and emitted:

text

0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844.Try --where class=bicycle_parking before concluding none exist.

0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844.Try --where class=bicycle_parking before concluding none exist.

result: c-wrong-column fell from 2 BEFORE to 0 AFTER.

Design implication:
For schema-heavy CLIs, a zero result should trigger cheap local diagnostics:

is the value known in another column?
is the field/value pair valid?
can the CLI suggest the corrected filter?

Strengthening experiment:
Generalize beyond class/subtype to other common field confusions and rerun on a larger matched subset.

Weakening experiment:
If broader hints increase wrong answers by suggesting semantically related but wrong fields, the rule must require high-confidence exact value matches only.

Plausible but not yet proven principles

3. Never return an unexplained zero

Statement: A zero result should distinguish “none exist” from “your query probably missed the data.”

Addresses: Class C / c-vocabulary, c-unknown, c-wrong-column, c-wrong-type.

Evidence:
c-unknown is tied for the largest remaining subtype bucket: 25 failures. Examples include counts for restaurant categories such as indonesian_restaurant, mongolian_restaurant, himalayan_restaurant returning empty with no conclusive probe.

Design implication:
A zero response should include diagnostic metadata:

filter values recognized/unrecognized,
field validity,
searched type,
nearby known values,
suggested discovery commands.

Strengthening experiment:
Implement a “zero diagnostics” hint for count: validate field/value/type and emit safe next probes. Rerun on c-unknown-heavy questions.

Weakening experiment:
If diagnostics are noisy or agents over-follow weak suggestions, distinguish high-confidence corrections from exploratory suggestions.

4. Name the fix, not just the problem

Statement: A CLI error should give the next command or corrected argument, not merely report invalid input.

Addresses: Class A vs B; recovery quality.

Evidence:
Taxonomy treats guided errors as B and near-free because they enable self-recovery. Class A remains high: 38 failures. Examples include usage-only failures such as:

with no actionable recovery path.

Design implication:
Usage text is not enough. Errors should include:

offending argument,
why it failed,
one or more valid alternatives,
exact command template to retry.

Strengthening experiment:
Pick a frequent A failure, add a concrete try ... recovery line, and measure A→B conversion plus whether the agent uses it.

Weakening experiment:
If agents ignore recovery lines, classify as F/ignored-hint and test whether placement/format matters.

5. Make discovery first-class

Statement: Agents should not need to guess, grep, or raise arbitrary limits to discover valid taxonomy values.

Addresses: c-truncated, c-vocabulary, c-unknown, token/wall-clock waste.

Evidence:
Truncation failures often arose from category discovery. The agent treated capped categories --top N output as complete. The confirmed truncation hint reduces one symptom, but not the deeper need for targeted search.

Design implication:
Provide commands like:

categories --search bus
schema values --field class
explain-filter class=bicycle_parking
count --diagnose ...

Strengthening experiment:
Add categories --search or similar, update no prompts, rerun discovery-heavy questions and compare commands/tokens.

Weakening experiment:
If agents still use broad listing routes, prompt/skill guidance may be required in addition to CLI affordance.

6. If the data lives under another type or verb, name that route

Statement: A zero under one feature type should say when the same filter has results under another type.

Addresses: Class C / c-wrong-type.

Evidence:
Baseline has 3 c-wrong-type failures. Example:

count -t land_use --where class=beach returned 0
land returned 65 for the same filter.

Design implication:
For multi-verb CLIs, validate likely type mismatches and say:

Strengthening experiment:
Implement type-sweep hint for zero counts with budget limits; run on beach-accessibility-malta and residential-share-cambridge.

Weakening experiment:
If type sweeps are too expensive or produce misleading alternatives, restrict to taxonomy-known exact matches.

7. Confirm resolved entities

Statement: When resolving place names or ambiguous qualifiers, the CLI should echo the resolved entity including region/country.

Addresses: Class C / c-wrong-entity candidate; evaluator false-positive risk.

Evidence:
Original c-wrong-entity counts were polluted by evaluator ambiguity: Cambridge, MA and Malta, MT can be US state abbreviations, not ISO country codes. This makes the principle plausible but not yet proven by corrected counts.

Design implication:
A CLI should make geocoding resolution explicit:

Strengthening experiment:
Create a paired ambiguous-entity subset with known cases and add explicit resolution echo.

Weakening experiment:
If no real attempts fail after corrected entity probing, keep this as defensive design, not confirmed principle.

8. Emit progress or offer cheaper routes for long operations

Statement: Long-running commands should show progress or suggest a narrower command.

Addresses: Class D / degenerate route; wall-clock waste.

Evidence:
Baseline has 3 D failures. Examples include where Cambridge, MA --geometry taking a degenerate route. docs/plan.md cites a 19m42s silent operation that looks like a hang.

Design implication:
Long operations should:

print progress on stderr,
estimate scope,
suggest bounding/filtering,
provide --dry-run/--estimate.

Strengthening experiment:
Add progress/estimate to one D route and measure abandonment, wall-clock, and final correctness.

Weakening experiment:
If agents do not abandon long commands in current harness or progress does not change behavior, classify as human-friendly more than agent-critical.

Evaluator and instrumentation principles

9. Silent wrong must be a first-class failure

Statement: A successful exit with misleading empty/plausible output can be worse than a crash.

Addresses: Class C generally.

Evidence:
Baseline found 55 class C failures that old scoring would under-detect. This restored headroom in an otherwise saturated exam.

Design implication for evaluators:
Do not equate exit 0 with success. Run differential probes after the agent finishes.

Experiment to strengthen:
Compare old score vs new class-C-aware score on same retained attempts and inspect previously “perfect” attempts.

10. Judge recoverability, not just errors

Statement: A CLI that refuses with a useful next step is behaving better than one that silently succeeds wrongly.

Addresses: Class B vs A/C; self-recovery score.

Evidence:
Scoring gives correctness/recoverability 60%, with self-recovery worth 20 points and guidance worth 12. Class B is near-free; class C and A are costly.

Design implication:
Evaluator should reward usable guidance and separately measure whether agents follow it.

Experiment to strengthen:
For the same invalid query, compare generic error vs exact correction and measure self-recovery rate.

11. Separate tool blame, environment failure, and agent-side failure

Statement: Quota/network failures and ignored hints must not be charged to the CLI.

Addresses: E and F.

Evidence:
Record-v2 supports attempt-level E verdicts. Clean record-v2 calls use JSON "class": null. F is recorded separately and not charged. Quota/environment failures are excluded.

Design implication:
Evaluator records need axes:

outcome,
blame,
recovery.

Class should be derived, not independently trusted.

Experiment to strengthen:
Inject controlled quota/network fixtures and ignored-hint traces into sabotage tests; verify scoring excludes/records correctly.

12. Probes must be audited and sabotage-tested

Statement: An evaluator for invisible failures must prove it catches known invisible failures and must expose probe evidence.

Addresses: evaluator false positives/negatives.

Evidence:
The MA/MT correction shows evaluator false findings are a first-class risk. The plan requires sabotage fixtures per class/subtype.

Design implication:
Every C label should include:

probe kind,
command run,
result,
conclusive flag,
evidence text.

Experiment to strengthen:
Add sabotage fixtures for each C subtype and one known false-positive ambiguity case.

Anti-principles / traps

1. “Exit 0 means the CLI helped”

Trap: agents convert empty success into confident wrong answers.
Counter-principle: exit 0 with empty output requires explanation or diagnostics.

2. “Terse human output is good agent output”

Trap: humans can infer truncation, ambiguity, or suspicious zeroes; agents often cannot.
Counter-principle: agents need explicit state: complete/truncated, resolved entity, recognized fields, valid values.

3. “Just print more help”

Trap: broad help increases token cost and may not name the local fix.
Counter-principle: give contextual, command-specific recovery advice.

4. “Optimize the benchmark by editing the judge”

Trap: Arm C correction showed optimizer must not change evaluator/yardstick.
Counter-principle: evaluator is measurement infrastructure; editable surface must exclude eval fixtures and scoring.

5. “Unknown means clean”

Trap: c-unknown is not proof the CLI behaved well; it is instrumentation debt.
Counter-principle: unresolved empty results stay suspicious until probes explain them.

6. “Hints automatically solve the problem”

Trap: a hint can be ignored.
Counter-principle: measure self-recovery and record ignored hints as F/agent-side, not as CLI blame.

Next experiments to strengthen the framework

Zero-result diagnostics experiment
Add high-confidence diagnostics for empty count results. Target c-unknown and c-vocabulary.
Wrong-type hint experiment
When same filter has rows under another feature type, name the correct type. Target c-wrong-type.
A→B recovery experiment
Pick common usage/raw-error failures and add exact retry commands. Measure A reduction and self-recovery.
Discovery command experiment
Add targeted category/value search, e.g. categories --search. Measure token/call reduction and final correctness.
Entity resolution echo experiment
Echo resolved place with country/region in JSON and stderr. Use deliberately ambiguous locations.
Progress/estimate experiment
Add progress or cost estimates to known degenerate long routes. Measure abandonment and wall-clock.
Full-bank replication of confirmed hints
Rerun truncation and wrong-column candidates over all 30 questions × 2 repeats to test generality.
Instruction-vs-tool lever comparison
For one principle, test tool-side hint vs skill.md instruction with same subset. This will clarify whether agent-friendliness belongs in the CLI or in agent guidance.

Some of these are botmap specific, if I have to make general claims, then I may have to experiment the same things on a different CLI

Yes. Right now the strongest claims are botmap-grounded, not universal. You can still phrase them generally, but only as candidate general principles confirmed in one CLI testbed.

A safe framing:

In botmap, agent failures concentrated around ambiguous successful outputs: silent truncation, unexplained zeroes, and field/type confusion. Paired interventions show that making these states explicit improved agent recovery on matched subsets. These are plausible general CLI principles, but cross-CLI replication is needed before claiming they hold broadly.

How to separate claim levels

General enough to claim cautiously now

These are not really map-specific:

Do not silently truncate output.
- General pattern: capped list treated as complete.
- Botmap evidence: c-truncated 13 → 5 after hint.
- Likely applies to any CLI with --limit, --top, pagination, search results.
Do not return ambiguous zeroes without diagnostics.
- General pattern: success + empty output can mean “none exist” or “bad query.”
- Botmap evidence: many class C empty results.
- Needs paired proof, but the mechanism is general.
Name the recovery action, not just the error.
- General pattern: agents recover better from exact retry commands than prose usage.
- Botmap evidence mostly taxonomy/scoring design; still needs direct paired A→B experiment.
Expose what the tool resolved/interpreted.
- General pattern: agents need to know how CLI parsed ambiguous input.
- Botmap-specific example: place/entity resolution.
- General analogue: path glob expansion, inferred project, selected account/region, selected cluster, default namespace.
Separate tool failure from environment and agent misuse in evaluation.
- This is evaluator-general, not botmap-specific.

Botmap-specific until replicated

These should be worded as examples, not universal laws:

class vs subtype confusion.
feature type mismatch: land_use vs land.
Overture taxonomy discovery.
geographic entity ambiguity like Cambridge, MA.
category search/discovery for map POIs.

The generalized version is:

When the CLI has a structured schema, it should validate field/value compatibility and surface likely mismatches.

But the evidence is currently from botmap’s schema.

A good next phase

Run a small replication study on one or two different CLIs with analogous failure modes.

Good candidate CLIs:

CLI type	Analogous failures
`kubectl`	namespace/context ambiguity, empty resource lists, wrong resource type
`gh`	repo/owner resolution, pagination, search limits, auth/env failures
`aws`	region/account defaults, pagination truncation, wrong service/resource fields
`jq`/`sqlite3` wrapper	wrong column, empty query, schema discovery
`ripgrep`/`fd`	ignored globs, hidden files, truncation/limits, no matches

For each, reuse the same structure:

Give agents plain-English tasks.
Record shell traces.
Classify failures with the same taxonomy.
Pick one high-frequency failure.
Add a CLI-side or instruction-side intervention.
Run paired BEFORE/AFTER.

Stronger general research question

Instead of “what makes botmap agent-friendly?”:

Which CLI behaviours help AI agents detect and recover from incorrect assumptions during command-line task solving?

Then botmap is Study 1, and another CLI becomes Study 2.

Best wording for current memo

Use:

confirmed in botmap

not:

confirmed universally

Use:

candidate general principle

not:

law of agent-friendly CLI design

Use:

cross-CLI replication needed

not:

proven for CLIs

That keeps the claims rigorous without losing the useful abstraction.

what about traces cli?

>find .. -maxdepth 3 -iname '*trace*' -o -iname '*traces*' | head -50 && pwd && ls

../autoresearch/autoresearch/trace.py
/Users/priyangapkini/nilenso/ai-playground/autoresearch
autoresearch
autoresearch.code-workspace
docs
experiments
pyproject.toml
README.md
tests
TODO.md
tools
uv.lock

>rg -n "traces? cli|trace" README.md docs autoresearch tools tests -g '*.md' -g '*.py' | head -80

tests/test_evaluator_agenteval.py:9:from autoresearch.trace import Call, Transcript, Usage
docs/agent-friendly-cli.md:5:trace; a property is not confirmed until a paired BEFORE/AFTER run shows the
docs/agent-friendly-cli.md:22:BEFORE      measured trace of an agent defeated by its absence
docs/agent-friendly-cli.md:24:AFTER       measured trace with the change in place
docs/agent-friendly-cli.md:93:            `bus-stops-cambridge`, 2 repeats each), the BEFORE trace had:
docs/agent-friendly-cli.md:131:            `residential-share-cambridge`, 2 repeats each), the BEFORE trace had:
docs/agent-friendly-cli.md:163:traces and later paired Phase 6 experiments.
autoresearch/smoke.py:123:        from .trace import parse_calls
tests/test_agenteval_taxonomy.py:38:def test_network_traceback_is_attempt_environment_shape_not_tool_crash():
README.md:201:| `trace.py`, `taxonomy.py` | read the logs; put a name to each failure |
README.md:215:work that way — it reads execution traces and writes its own changes — so
README.md:227:run scores zero immediately instead of burning $0.50 on 30 tracebacks — and
tests/test_probe.py:13:from autoresearch.trace import Call
tests/test_record_v2.py:6:from autoresearch.trace import Call, Transcript
autoresearch/score.py:15:from .trace import Call, Transcript
autoresearch/config.py:165:# network dropped is recorded by the taxonomy as `traceback` -- the same label
autoresearch/config.py:403:                f"({rate:.0%}). Failed calls are recorded as `traceback`, the "
autoresearch/optimize.py:66:- crashing with a raw Python traceback.
autoresearch/runner.py:21:from .trace import parse_calls, parse_transcript
tests/test_scoring.py:15:from autoresearch.trace import Call, Transcript, Usage
tests/test_scoring.py:48:            == "traceback"
tests/test_scoring.py:495:    `traceback` — the same label a genuine bug gets. So the network becomes the
docs/plan.md:46:BEFORE      the measured trace of an agent defeated by its absence
docs/plan.md:55:AFTER       the measured trace of the same question, same conditions,
docs/plan.md:121:**STATUS: hypothesis** — we have the BEFORE trace for each and no AFTER for any.
docs/plan.md:281:| **A** Hard failure | `error` + `unguided` | Refused, agent stuck. Raw traceback. | heavy penalty |
docs/plan.md:488:   traces. It has the deepest trace knowledge and `failure_dataset.yaml` already
docs/plan.md:519:                                                 change it, re-run, compare traces
docs/plan.md:526:2. the BEFORE traces for every candidate property.
docs/plan.md:530:conditions, and compare traces. Commands burned, turns taken, whether the agent
docs/plan.md:646:  traces (`failure_dataset.yaml` already maps 26 questions to their defects)
docs/plan.md:665:*Output:* the class histogram, and a BEFORE trace for every candidate property.
docs/plan.md:679:the same conditions, compare traces. Arm A's three candidates are the first
autoresearch/taxonomy.py:7:traceback also contains the word "error".
autoresearch/taxonomy.py:12:from .trace import Call
autoresearch/taxonomy.py:19:    "traceback",
autoresearch/taxonomy.py:45:        return "traceback"
autoresearch/agenteval/repair.py:15:from autoresearch.trace import Call
autoresearch/agenteval/repair.py:97:    from autoresearch.trace import Transcript
autoresearch/agenteval/enrich.py:13:from autoresearch.trace import parse_calls
autoresearch/agenteval/enrich.py:103:    from autoresearch.trace import parse_transcript
autoresearch/agenteval/sabotage.py:123:        id="a-raw-traceback-unguided",
autoresearch/agenteval/sabotage.py:126:        source="docs/plan.md class A: raw traceback/refusal left agent unguided",
autoresearch/agenteval/sabotage.py:132:            evidence="unhandled tool traceback, no retry instruction",
autoresearch/agenteval/sabotage.py:229:        source="baseline-noise-run1 beach-accessibility-malta/malta-highways traces: Malta, MT vs Malta country ambiguity",
docs/agent-failure-taxonomy.md:154:`outcome=error, recovery=unguided`. Raw traceback, unclassified error.
autoresearch/agenteval/record.py:11:from autoresearch.trace import Call, Transcript
autoresearch/agenteval/probe.py:15:from autoresearch.trace import Call

Yes — a traces CLI could be a good second testbed, especially because it is not map/taxonomy-specific but still has structured-query failure modes similar to botmap.

It would test whether the principles generalize from “map data CLI” to observability / log-analysis CLI.

Why traces CLI is a strong candidate

A traces CLI likely has analogous agent failure modes:

General principle	Botmap example	Traces CLI analogue
Never silently truncate	`categories --top 100` omits values	trace search returns first N spans/events without saying there are more
Explain empty results	`count` returns 0 for wrong filter	`traces find --where status=error` returns 0 because field is actually `level` / `outcome`
If value exists elsewhere, say where	`subtype=bicycle_parking` should be `class=bicycle_parking`	`model=gpt-4` exists under `metadata.model`, not `span.name`
Confirm resolved input	`Cambridge, MA` resolved ambiguously	time range, session id prefix, run id prefix, service name resolved ambiguously
Make discovery first-class	no easy category search	need `schema`, `fields`, `values`, `examples`, `search-fields`
Avoid degenerate routes	broad geometry call	dumping all traces / huge JSONL grep instead of indexed query

So it could replicate the same underlying pattern: agents fail when the CLI hides query interpretation and result completeness.

Why it may be even better than botmap

Traces/log CLIs are closer to how coding agents actually work:

agents inspect logs,
search by partial IDs,
filter by status/error/tool/model,
correlate events,
handle JSON/JSONL,
deal with large outputs,
need summarization and drill-down.

That makes it a useful generalization target for “agent-friendly developer tools,” not just geospatial CLIs.

Main caveat

If the traces CLI is part of this autoresearch harness, be careful: it may be too close to the evaluator domain.

Good use:

Use traces CLI as an independent subject CLI that agents must operate.

Risky use:

Let optimizer/evaluator modify traces CLI or judge itself using its own trace abstractions.

So I’d keep the same guardrail as botmap:

evaluator read-only,
traces CLI treated as the subject,
no editing the judge to improve the score.

Good experiment design

Pick 6–10 plain-English tasks like:

“Find the first failed tool call in run X.”
“Which command caused the longest delay?”
“How many attempts ended due to quota?”
“Find all calls where the agent got zero results but continued.”
“Which run used WebSearch before any CLI command?”
“Show the stderr for the failed command in attempt Y.”
“Compare token usage between two attempts.”
“Find traces where a hint was emitted but ignored.”

Then classify failures using the same taxonomy:

C-truncated: trace list capped; agent misses later failure.
C-wrong-column: filters wrong field, e.g. status=failed vs outcome=error.
C-wrong-type: searches spans when data is in events/messages.
C-unknown: zero trace results unexplained.
A: invalid query/raw traceback/no recovery.
D: dumps entire trace file and greps huge output.

Best first paired experiments

I’d start with these because they map cleanly to current botmap findings:

1. Truncation notice

If traces list/search returns capped output, emit:

text

Showing 100 of 2,341 events. This result is truncated; rerun with --limit 2341or add --after/--before/--filter before concluding no matching event exists.

Showing 100 of 2,341 events. This result is truncated; rerun with --limit 2341or add --after/--before/--filter before concluding no matching event exists.

This directly replicates the confirmed botmap principle.

2. Field/value correction

If query returns zero but value exists in another field:

text

0 results for status='quota_exhausted', but outcome='quota_exhausted' appearsin 17 events. Try --where outcome=quota_exhausted.

0 results for status='quota_exhausted', but outcome='quota_exhausted' appearsin 17 events. Try --where outcome=quota_exhausted.

This replicates the confirmed wrong-column principle.

3. Schema discovery

Add or test:

bash

traces schematraces values --field outcometraces fields --search quotatraces explain-filter outcome=quota_exhausted

traces schematraces values --field outcometraces fields --search quotatraces explain-filter outcome=quota_exhausted

This tests the broader “make discovery first-class” principle.

Claim you could make after botmap + traces

If both show the same result, you can strengthen the general claim to:

Across a geospatial data CLI and a trace-inspection CLI, agent failures clustered around hidden query semantics: silent truncation, ambiguous zeroes, and unexposed field/value mismatches. In both settings, surfacing completeness and correction hints improved recoverability.

That would be much stronger than botmap alone.

So yes: traces CLI is probably one of the best next CLIs, provided you treat it as a separate subject and don’t let it collapse into the evaluator itself.

SYNTHESIS REQUEST: answer the core research question.

Stop any experimental execution. Do not run paid work. Do not run GEPA. Do not run autoresearch.runner. Do not modify implementation code.

Question to answer: What are the general principles from the experiments so far that help answer: what makes a CLI agent-friendly?

Use evidence from the autoresearch/botmap experiment, especially:

docs/agent-friendly-cli.md
experiments/paired/categories-truncation-hint/result.md
experiments/paired/count-wrong-column-hint/result.md
experiments/runs/agenteval-measurement-3009509/agenteval-summary-with-retries.json
experiments/arm-d/PLAN.md if relevant
your own arm/session observations

Your answer should separate:

Principles actually supported by paired evidence.
Principles suggested by baseline traces but not yet experimentally confirmed.
Evaluator/instrumentation principles we learned.
Anti-patterns: things that make a CLI hostile to agents.
What a CLI author should do differently tomorrow.

For each principle include:

concise name
statement
evidence/failure class
level of confidence: confirmed / provisional / hypothesis
design implication

Write your answer to one file:

pi-arm-a: experiments/arm-a/GENERAL-PRINCIPLES.md
pi-arm-b: experiments/arm-b/GENERAL-PRINCIPLES.md
pi-arm-c: experiments/arm-c/GENERAL-PRINCIPLES.md
pi-arm-d: experiments/arm-d/GENERAL-PRINCIPLES.md
derivation: experiments/derivation/GENERAL-PRINCIPLES.md
blog-ideation: experiments/blog-ideation/GENERAL-PRINCIPLES.md

Create the directory if needed. After writing, reply with the file path and top 5 principles only.

Wrote: experiments/derivation/GENERAL-PRINCIPLES.md

Top 5 principles:

Never silently truncate output.
If a value exists elsewhere, say where.
Do not return an unexplained zero.
Name the fix, not just the problem.
Make discovery first-class.