goldfishandswedishfish shared

Reviewed accuracy estimation in Seahaven docs

Agent

Claude Code

Model

Claude Opus 5.5

Started

Oct 14, 12:48 PM

Messages28

Downloads0

Summary

Closed, Completed

Understand
Reviewed CONTRIBUTING.md content.
Understand
Understand
Checked for commands estimating simulation accuracy.
Understand
Verify
Confirmed no AI assessment or accuracy estimation exists.
Verify

Review this: https://github.com/Kiln-AI/Seahaven/blob/main/.github/CONTRIBUTING.md is there a command that helps the user understand/estimate how accurate the simulated world is. Or at least the AIs take of that accuracy

Cloning the repo is simpler than searching through the API.

No. Neither CONTRIBUTING.md nor any Seahaven command estimates how accurate the simulated world is, and nothing asks an AI to judge it either.

What CONTRIBUTING.md covers: it's only for people working on the framework itself. It explains the uv/Python 3.14 setup, the CI checks they must pass (ruff, ty, pytest, the licence check), and how to work with the two conflicting environments (serve and mcp). It doesn't describe anything for world users.

The full command list (I cloned the repo and read src/seahaven/cli/):

Command	What it does
`new`	Creates a new world project
`check`	Runs every lint over the world
`fixture list / freeze / fork`	Manages saved starting states
`serve`	Runs the OpenEnv server and web console
`mcp`	Serves the world to an MCP client
`hub`	Adds the files needed to publish to a hub
`docs`	Prints the path to the bundled docs

Why check doesn't do this: it sounds like the closest match, but it tests whether the world follows Seahaven's own rules. Examples are bad SQL in the schema or a package with no world in it, and each finding comes with a code and a fix. It never compares the world to the real system it's copying.

How the project treats accuracy: it's left to whoever builds the world. The README promises worlds "realistic enough that the agent can't tell the difference," but there's no way to measure that. The docs only mention "infidelity" as a tradeoff the author makes knowingly (docs/authoring.md:592, docs/composition.md:186).

What you can do instead:

Check it by hand: seahaven serve opens a web console at /console where you can call tools and look at the data. seahaven mcp lets you poke at the world from Claude or an editor.
Get an AI's opinion: point a coding agent at the world's tools and schema plus the real API's docs, and ask it to list where they differ: missing fields, error formats, pagination, edge cases. That's the closest thing to the "AI's take" you're after, but you'd have to set it up yourself.
Ask the maintainers: CONTRIBUTING asks people to open an issue before starting anything bigger than a bug fix. A "fidelity report" command would count, so an issue is the place to propose one.

Help me understand how this solution would be built?

A fidelity command would compare the world against a reference for the real system. Seahaven already gives you the world's side through code you can inspect. The work is getting the real system's side and deciding how to compare the two.

What the code already gives you

world.tools (src/seahaven/world.py:224) maps each tool name to a Tool. Each Tool has a name, a description, and a schema, which is the JSON Schema of its parameters, built from a pydantic model (src/seahaven/tool.py:75-79).
world.instance(fixture) with .call() and .state() (src/seahaven/handles.py) runs real tool calls against a throwaway copy of the world and shows exactly what changed.
discover() (src/seahaven/cli/__init__.py:182) finds and imports the world, the same way check does.
Finding (src/seahaven/lint/__init__.py:59) is the output format check uses: a code, a severity, a location and a fix. A fidelity command could reuse it, so its output would read like check's.

Three checks, from cheapest to most expensive

1. Compare the tool list (no AI needed). Take the world's tools and compare them to a reference: an OpenAPI spec, or the tools/list output from the real service's MCP server. Report:

tools that are missing or extra
parameters that are missing, renamed, or a different type
differences in which fields are required

The results are exact and repeatable, and they give you an honest number: "142 of 155 operations match the reference." Stripe World says it exposes "the same tools as Stripe's own MCP server," so it's a ready-made test case.

2. Compare behavior. Make the same calls against the world and against the real system's sandbox (for example, Stripe test mode), then compare:

the shape of the responses
error types and status codes
pagination
edge cases, like an ID that doesn't exist or a duplicate create

Compare shapes, not values, because the two have different data. Record the real responses once and save them, so later runs work offline and give the same results every time. That matters to this project: the README lists reproducibility as a main feature.

3. Ask an AI judge (the "AI's take" you mentioned). Some things can't be checked mechanically:

Is the fixture data realistic?
Are the error messages worded like the real ones?
Is behavior that's described in prose docs, but not in any spec, actually implemented?

Give a model the real system's docs, the world's tool descriptions and schemas, and sample outputs from check 2. Ask for structured findings, each one quoting the docs passage and the world output it's based on. Treat the result as an opinion. Requiring a quote for every finding is what keeps it from making things up. I wouldn't report a single "87% accurate" score here, because the number would look precise without meaning much. Per-tool numbers from checks 1 and 2 are defensible.

Where it fits in the code

As a core command, it would be a new src/seahaven/cli/fidelity.py with an add_parser function, added to the tuple at src/seahaven/cli/__init__.py:135, using discover() the way check does.

But checks 2 and 3 need network access, real-service credentials and model API keys. check deliberately doesn't (it's offline and gives the same output every time), and new dependencies have to be permissively licensed. So it probably fits better as a separate extension package, like extensions/seahaven-xmlrpc. You could also build only check 1 into core and keep the rest outside.

Suggested first version

Build only check 1:

It would load the world, load the reference tool list, compare names and parameter schemas, and print the differences in the same format as check. That's about a day's work, it needs no API keys, and it tests whether the idea is worth pursuing. Bring it to the maintainers in an issue before opening a PR, since CONTRIBUTING asks for that on anything bigger than a bug fix.

I can prototype check 1 against Stripe World if you'd like. Just tell me where you want the files. I'd suggest somewhere outside the repo, like your Desktop.

So: in short -- there's no way, currently, to help the user reason about the fidelity/accuracy of the world that's been spun up?

That's right. Nothing built in measures or estimates how faithful a world is to the real system, and nothing asks an AI to judge it. I went through all the CLI commands and searched the docs and source, and found nothing that does this.

What you have is ways to inspect the world yourself, not ways to judge it:

seahaven serve runs a web console where you can call tools and look at the data.
seahaven mcp lets you explore the world from Claude or an editor.
inst.state() shows exactly what each call changed.
The world's own pytest tests show what the author chose to check.

So you can get a feel for the world by using it, but the comparison to the real system happens in your head, or in tests the author wrote. Seahaven doesn't produce any report or score for it.