Derived agent-friendly CLI principles
Public
Derived agent-friendly CLI principles
Agent
Pi
Model
GPT-5.5
Started
Aug 24, 8:41 AM
Messages49
Downloads0
Workspace
Reviewed key documentation and experiment results.
UnderstandSynthesized general principles for agent-friendly CLI.
BuildDocumented top principles in a summary file.
EditYou are a fresh Pi session named DERIVATION. Your job is not implementation and not blogging style. Your job is to derive principles for making an agent-friendly CLI from the autoresearch/botmap experiments so far.
Work in repo: /Users/priyangapkini/nilenso/ai-playground/autoresearch
Do not run paid experiments. Do not modify code unless explicitly asked. Read files as needed.
Key files to read first:
Experiment goal:
Derive what makes a CLI friendly to AI agents. The concrete testbed is botmap, a CLI over Overture map data. Agents get plain-English map questions and must discover/compose the right commands via shell use. The output we want is principles grounded in measured traces, not vibes.
Important constraints/context:
botmap/evals/.autoresearch/agenteval/ and is now the intended judge."class": null.Evaluator/taxonomy summary:
Baseline/BEFORE measurement:
Cambridge, MA and Malta, MT can be US state abbreviations, not ISO country codes. False evaluator findings are a first-class risk.Paired experiments completed:
00bff1a Warn when categories output is truncatedbotmap categories output is capped, emit stderr hint while preserving JSON stdout.c-truncated failures dropped from 13 BEFORE to 5 AFTER.7c794ff Hint when count filter uses class and subtype wrongcount returns zero for class=X or subtype=X, test the swapped field and emit correction on stderr.0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844. Try --where class=bicycle_parking before concluding none exist.c-wrong-column fell from 2 BEFORE to 0 AFTER on matched subset.Current arms/runs, for context only:
experiments/runs/arm-a-new-evaluator-count-zero-hint-9a2496d/.botmap/data/skill.md; run experiments/runs/prompt-3009509-1787544952/.experiments/runs/tool-3009509-1787550419/.evals/*, tests/eval_fixtures/*, tests/test_eval_*.Your derivation task:
Start by reading docs/agent-friendly-cli.md and the two paired result files, then produce your derivation memo.
An agent-friendly CLI is not mainly “feature-rich” or “pleasant for humans.” It is observable, self-describing, and recovery-oriented under shell composition. The botmap traces show agents often fail not because the command is impossible, but because the CLI returns an ambiguous success, a silent zero, a capped list, or a generic error that gives no next move.
For humans, terse output is often acceptable because humans bring domain knowledge and can notice suspicious absence. For agents, absence is data. A successful empty result or capped list is easily incorporated into a confident wrong answer unless the CLI makes uncertainty explicit and tells the agent how to verify or recover.
Evidence base read:
docs/agent-friendly-cli.mddocs/plan.mdexperiments/orchestrator/HANDOVER.mdexperiments/paired/categories-truncation-hint/result.mdexperiments/paired/count-wrong-column-hint/result.mdautoresearch/agenteval/contract.pytaxonomy.pyscore.pyexplain.pyprobe.pyexperiments/runs/agenteval-measurement-3009509/agenteval-summary-with-retries.jsonBaseline corrected distribution:
c-truncated 25, c-unknown 25, c-wrong-type 3, c-wrong-column 2Statement: If output is capped, the CLI must say it is capped and name the exact way to get complete output.
Addresses: Class C / c-truncated.
Evidence:
Baseline showed many category discovery calls where output length equalled --top, and a limit-raise probe found more rows. Example: categories -t place --in Cambridge, MA --top 100 returned 100 rows; raised limit returned 980.
Paired experiment:
bike-parking-coverage, basic-category-rollup, bus-stops-cambridge, 2 repeats each[botmap] Showing top N of TOTAL categories. This list is truncated; rerun with`--top TOTAL` or a larger --top before concluding a category is absent.c-truncated fell from 13 BEFORE to 5 AFTER.Design implication:
Any CLI producing lists must expose:
Strengthening experiment:
Run the truncation hint across the full question bank and measure not only subtype reduction but final-answer correctness, recovery commands, tokens, and wall-clock.
Weakening experiment:
If full-bank paired runs show agents still make absence conclusions despite the hint, the principle may need refinement: hints must be machine-readable or included in JSON metadata, not just stderr prose.
Statement: When a zero result uses a real value in the wrong field, the CLI should name the field where the value exists and provide the corrected filter.
Addresses: Class C / c-wrong-column.
Evidence:
Baseline examples:
subtype=bicycle_parking returned 0, but class=bicycle_parking returned 1,844.subtype=government returned 0, but class=government returned 23.Paired experiment:
bike-parking-coverage, residential-share-cambridge, 2 repeats eachbotmap count tested swapped class/subtype field on zero and emitted:0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844.Try --where class=bicycle_parking before concluding none exist.c-wrong-column fell from 2 BEFORE to 0 AFTER.Design implication:
For schema-heavy CLIs, a zero result should trigger cheap local diagnostics:
Strengthening experiment:
Generalize beyond class/subtype to other common field confusions and rerun on a larger matched subset.
Weakening experiment:
If broader hints increase wrong answers by suggesting semantically related but wrong fields, the rule must require high-confidence exact value matches only.
Statement: A zero result should distinguish “none exist” from “your query probably missed the data.”
Addresses: Class C / c-vocabulary, c-unknown, c-wrong-column, c-wrong-type.
Evidence:
c-unknown is tied for the largest remaining subtype bucket: 25 failures. Examples include counts for restaurant categories such as indonesian_restaurant, mongolian_restaurant, himalayan_restaurant returning empty with no conclusive probe.
Design implication:
A zero response should include diagnostic metadata:
Strengthening experiment:
Implement a “zero diagnostics” hint for count: validate field/value/type and emit safe next probes. Rerun on c-unknown-heavy questions.
Weakening experiment:
If diagnostics are noisy or agents over-follow weak suggestions, distinguish high-confidence corrections from exploratory suggestions.
Statement: A CLI error should give the next command or corrected argument, not merely report invalid input.
Addresses: Class A vs B; recovery quality.
Evidence:
Taxonomy treats guided errors as B and near-free because they enable self-recovery. Class A remains high: 38 failures. Examples include usage-only failures such as:
Usage: python -m botmap count [OPTIONS]with no actionable recovery path.
Design implication:
Usage text is not enough. Errors should include:
Strengthening experiment:
Pick a frequent A failure, add a concrete try ... recovery line, and measure A→B conversion plus whether the agent uses it.
Weakening experiment:
If agents ignore recovery lines, classify as F/ignored-hint and test whether placement/format matters.
Statement: Agents should not need to guess, grep, or raise arbitrary limits to discover valid taxonomy values.
Addresses: c-truncated, c-vocabulary, c-unknown, token/wall-clock waste.
Evidence:
Truncation failures often arose from category discovery. The agent treated capped categories --top N output as complete. The confirmed truncation hint reduces one symptom, but not the deeper need for targeted search.
Design implication:
Provide commands like:
categories --search busschema values --field classexplain-filter class=bicycle_parkingcount --diagnose ...Strengthening experiment:
Add categories --search or similar, update no prompts, rerun discovery-heavy questions and compare commands/tokens.
Weakening experiment:
If agents still use broad listing routes, prompt/skill guidance may be required in addition to CLI affordance.
Statement: A zero under one feature type should say when the same filter has results under another type.
Addresses: Class C / c-wrong-type.
Evidence:
Baseline has 3 c-wrong-type failures. Example:
count -t land_use --where class=beach returned 0land returned 65 for the same filter.Design implication:
For multi-verb CLIs, validate likely type mismatches and say:
0 rows for type land_use, but land has 65 rows for class=beach. Try ...Strengthening experiment:
Implement type-sweep hint for zero counts with budget limits; run on beach-accessibility-malta and residential-share-cambridge.
Weakening experiment:
If type sweeps are too expensive or produce misleading alternatives, restrict to taxonomy-known exact matches.
Statement: When resolving place names or ambiguous qualifiers, the CLI should echo the resolved entity including region/country.
Addresses: Class C / c-wrong-entity candidate; evaluator false-positive risk.
Evidence:
Original c-wrong-entity counts were polluted by evaluator ambiguity: Cambridge, MA and Malta, MT can be US state abbreviations, not ISO country codes. This makes the principle plausible but not yet proven by corrected counts.
Design implication:
A CLI should make geocoding resolution explicit:
{"query":"Cambridge, MA", "resolved":"Cambridge, Massachusetts, US", ...}Strengthening experiment:
Create a paired ambiguous-entity subset with known cases and add explicit resolution echo.
Weakening experiment:
If no real attempts fail after corrected entity probing, keep this as defensive design, not confirmed principle.
Statement: Long-running commands should show progress or suggest a narrower command.
Addresses: Class D / degenerate route; wall-clock waste.
Evidence:
Baseline has 3 D failures. Examples include where Cambridge, MA --geometry taking a degenerate route. docs/plan.md cites a 19m42s silent operation that looks like a hang.
Design implication:
Long operations should:
--dry-run/--estimate.Strengthening experiment:
Add progress/estimate to one D route and measure abandonment, wall-clock, and final correctness.
Weakening experiment:
If agents do not abandon long commands in current harness or progress does not change behavior, classify as human-friendly more than agent-critical.
Statement: A successful exit with misleading empty/plausible output can be worse than a crash.
Addresses: Class C generally.
Evidence:
Baseline found 55 class C failures that old scoring would under-detect. This restored headroom in an otherwise saturated exam.
Design implication for evaluators:
Do not equate exit 0 with success. Run differential probes after the agent finishes.
Experiment to strengthen:
Compare old score vs new class-C-aware score on same retained attempts and inspect previously “perfect” attempts.
Statement: A CLI that refuses with a useful next step is behaving better than one that silently succeeds wrongly.
Addresses: Class B vs A/C; self-recovery score.
Evidence:
Scoring gives correctness/recoverability 60%, with self-recovery worth 20 points and guidance worth 12. Class B is near-free; class C and A are costly.
Design implication:
Evaluator should reward usable guidance and separately measure whether agents follow it.
Experiment to strengthen:
For the same invalid query, compare generic error vs exact correction and measure self-recovery rate.
Statement: Quota/network failures and ignored hints must not be charged to the CLI.
Addresses: E and F.
Evidence:
Record-v2 supports attempt-level E verdicts. Clean record-v2 calls use JSON "class": null. F is recorded separately and not charged. Quota/environment failures are excluded.
Design implication:
Evaluator records need axes:
Class should be derived, not independently trusted.
Experiment to strengthen:
Inject controlled quota/network fixtures and ignored-hint traces into sabotage tests; verify scoring excludes/records correctly.
Statement: An evaluator for invisible failures must prove it catches known invisible failures and must expose probe evidence.
Addresses: evaluator false positives/negatives.
Evidence:
The MA/MT correction shows evaluator false findings are a first-class risk. The plan requires sabotage fixtures per class/subtype.
Design implication:
Every C label should include:
Experiment to strengthen:
Add sabotage fixtures for each C subtype and one known false-positive ambiguity case.
Trap: agents convert empty success into confident wrong answers.
Counter-principle: exit 0 with empty output requires explanation or diagnostics.
Trap: humans can infer truncation, ambiguity, or suspicious zeroes; agents often cannot.
Counter-principle: agents need explicit state: complete/truncated, resolved entity, recognized fields, valid values.
Trap: broad help increases token cost and may not name the local fix.
Counter-principle: give contextual, command-specific recovery advice.
Trap: Arm C correction showed optimizer must not change evaluator/yardstick.
Counter-principle: evaluator is measurement infrastructure; editable surface must exclude eval fixtures and scoring.
Trap: c-unknown is not proof the CLI behaved well; it is instrumentation debt.
Counter-principle: unresolved empty results stay suspicious until probes explain them.
Trap: a hint can be ignored.
Counter-principle: measure self-recovery and record ignored hints as F/agent-side, not as CLI blame.
Zero-result diagnostics experiment
Add high-confidence diagnostics for empty count results. Target c-unknown and c-vocabulary.
Wrong-type hint experiment
When same filter has rows under another feature type, name the correct type. Target c-wrong-type.
A→B recovery experiment
Pick common usage/raw-error failures and add exact retry commands. Measure A reduction and self-recovery.
Discovery command experiment
Add targeted category/value search, e.g. categories --search. Measure token/call reduction and final correctness.
Entity resolution echo experiment
Echo resolved place with country/region in JSON and stderr. Use deliberately ambiguous locations.
Progress/estimate experiment
Add progress or cost estimates to known degenerate long routes. Measure abandonment and wall-clock.
Full-bank replication of confirmed hints
Rerun truncation and wrong-column candidates over all 30 questions × 2 repeats to test generality.
Instruction-vs-tool lever comparison
For one principle, test tool-side hint vs skill.md instruction with same subset. This will clarify whether agent-friendliness belongs in the CLI or in agent guidance.
Some of these are botmap specific, if I have to make general claims, then I may have to experiment the same things on a different CLI
Yes. Right now the strongest claims are botmap-grounded, not universal. You can still phrase them generally, but only as candidate general principles confirmed in one CLI testbed.
A safe framing:
In botmap, agent failures concentrated around ambiguous successful outputs: silent truncation, unexplained zeroes, and field/type confusion. Paired interventions show that making these states explicit improved agent recovery on matched subsets. These are plausible general CLI principles, but cross-CLI replication is needed before claiming they hold broadly.
These are not really map-specific:
Do not silently truncate output.
c-truncated 13 → 5 after hint.--limit, --top, pagination, search results.Do not return ambiguous zeroes without diagnostics.
Name the recovery action, not just the error.
Expose what the tool resolved/interpreted.
Separate tool failure from environment and agent misuse in evaluation.
These should be worded as examples, not universal laws:
class vs subtype confusion.land_use vs land.Cambridge, MA.The generalized version is:
When the CLI has a structured schema, it should validate field/value compatibility and surface likely mismatches.
But the evidence is currently from botmap’s schema.
Run a small replication study on one or two different CLIs with analogous failure modes.
Good candidate CLIs:
| CLI type | Analogous failures |
|---|---|
kubectl | namespace/context ambiguity, empty resource lists, wrong resource type |
gh | repo/owner resolution, pagination, search limits, auth/env failures |
aws | region/account defaults, pagination truncation, wrong service/resource fields |
jq/sqlite3 wrapper | wrong column, empty query, schema discovery |
ripgrep/fd | ignored globs, hidden files, truncation/limits, no matches |
For each, reuse the same structure:
Instead of “what makes botmap agent-friendly?”:
Which CLI behaviours help AI agents detect and recover from incorrect assumptions during command-line task solving?
Then botmap is Study 1, and another CLI becomes Study 2.
Use:
confirmed in botmap
not:
confirmed universally
Use:
candidate general principle
not:
law of agent-friendly CLI design
Use:
cross-CLI replication needed
not:
proven for CLIs
That keeps the claims rigorous without losing the useful abstraction.
what about traces cli?
Yes — a traces CLI could be a good second testbed, especially because it is not map/taxonomy-specific but still has structured-query failure modes similar to botmap.
It would test whether the principles generalize from “map data CLI” to observability / log-analysis CLI.
A traces CLI likely has analogous agent failure modes:
| General principle | Botmap example | Traces CLI analogue |
|---|---|---|
| Never silently truncate | categories --top 100 omits values | trace search returns first N spans/events without saying there are more |
| Explain empty results | count returns 0 for wrong filter | traces find --where status=error returns 0 because field is actually level / outcome |
| If value exists elsewhere, say where | subtype=bicycle_parking should be class=bicycle_parking | model=gpt-4 exists under metadata.model, not span.name |
| Confirm resolved input | Cambridge, MA resolved ambiguously | time range, session id prefix, run id prefix, service name resolved ambiguously |
| Make discovery first-class | no easy category search | need schema, fields, values, examples, search-fields |
| Avoid degenerate routes | broad geometry call | dumping all traces / huge JSONL grep instead of indexed query |
So it could replicate the same underlying pattern: agents fail when the CLI hides query interpretation and result completeness.
Traces/log CLIs are closer to how coding agents actually work:
That makes it a useful generalization target for “agent-friendly developer tools,” not just geospatial CLIs.
If the traces CLI is part of this autoresearch harness, be careful: it may be too close to the evaluator domain.
Good use:
Use traces CLI as an independent subject CLI that agents must operate.
Risky use:
Let optimizer/evaluator modify traces CLI or judge itself using its own trace abstractions.
So I’d keep the same guardrail as botmap:
Pick 6–10 plain-English tasks like:
Then classify failures using the same taxonomy:
status=failed vs outcome=error.I’d start with these because they map cleanly to current botmap findings:
If traces list/search returns capped output, emit:
Showing 100 of 2,341 events. This result is truncated; rerun with --limit 2341or add --after/--before/--filter before concluding no matching event exists.This directly replicates the confirmed botmap principle.
If query returns zero but value exists in another field:
0 results for status='quota_exhausted', but outcome='quota_exhausted' appearsin 17 events. Try --where outcome=quota_exhausted.This replicates the confirmed wrong-column principle.
Add or test:
traces schematraces values --field outcometraces fields --search quotatraces explain-filter outcome=quota_exhaustedThis tests the broader “make discovery first-class” principle.
If both show the same result, you can strengthen the general claim to:
Across a geospatial data CLI and a trace-inspection CLI, agent failures clustered around hidden query semantics: silent truncation, ambiguous zeroes, and unexposed field/value mismatches. In both settings, surfacing completeness and correction hints improved recoverability.
That would be much stronger than botmap alone.
So yes: traces CLI is probably one of the best next CLIs, provided you treat it as a separate subject and don’t let it collapse into the evaluator itself.
SYNTHESIS REQUEST: answer the core research question.
Stop any experimental execution. Do not run paid work. Do not run GEPA. Do not run autoresearch.runner. Do not modify implementation code.
Question to answer: What are the general principles from the experiments so far that help answer: what makes a CLI agent-friendly?
Use evidence from the autoresearch/botmap experiment, especially:
Your answer should separate:
For each principle include:
Write your answer to one file:
Create the directory if needed. After writing, reply with the file path and top 5 principles only.
Wrote: experiments/derivation/GENERAL-PRINCIPLES.md
Top 5 principles:
[botmap] Showing top N of TOTAL categories. This list is truncated; rerun with`--top TOTAL` or a larger --top before concluding a category is absent.0 rows for subtype='bicycle_parking', but class='bicycle_parking' returns 1,844.Try --where class=bicycle_parking before concluding none exist.Usage: python -m botmap count [OPTIONS]0 rows for type land_use, but land has 65 rows for class=beach. Try ...{"query":"Cambridge, MA", "resolved":"Cambridge, Massachusetts, US", ...}Showing 100 of 2,341 events. This result is truncated; rerun with --limit 2341or add --after/--before/--filter before concluding no matching event exists.0 results for status='quota_exhausted', but outcome='quota_exhausted' appearsin 17 events. Try --where outcome=quota_exhausted.traces schematraces values --field outcometraces fields --search quotatraces explain-filter outcome=quota_exhausted