goldfishandswedishfish shared

Reviewed accuracy estimation in Seahaven docs

Agent

Claude Code

Model

Claude Opus 5.5

Started

Oct 14, 12:48 PM

Messages28

Downloads0

Summary

Closed, Completed

Understand
Reviewed CONTRIBUTING.md content.
Understand
Understand
Checked for commands estimating simulation accuracy.
Understand
Verify
Confirmed no AI assessment or accuracy estimation exists.
Verify

No content

Review this: https://github.com/Kiln-AI/Seahaven/blob/main/.github/CONTRIBUTING.md is there a command that helps the user understand/estimate how accurate the simulated world is. Or at least the AIs take of that accuracy

>gh api repos/Kiln-AI/Seahaven/contents/.github/CONTRIBUTING.md -H "Accept: application/vnd.github.raw" 2>&1 | head -400

# Contributing to Seahaven

Thanks for your interest. Seahaven is early and the API is still moving, so open an issue before
starting anything larger than a bug fix.

## Setup

You need [uv](https://docs.astral.sh/uv/) and a final release of Python 3.14 or newer. Not a
release candidate: two locked dependencies break on the 3.14 rcs, and uv will not stop you from
syncing onto one.

```sh
uv python install 3.14
uv sync --extra serve
uv run python -V    # must be 3.14.0 or newer, with no "rc"
```

## Running the checks

CI runs these, and all of them must pass before you open a pull request:

```sh
uv run ruff format --check
uv run ruff check
uv run ty check -c 'src.exclude=["src/seahaven/mcp", "tests/test_mcp_server.py", "tests/test_mcp_process.py"]'
uv run pytest                              # the framework
uv run pytest worlds/projecttracker        # the reference world
uv run pytest extensions/seahaven-xmlrpc   # the example extension
uv run python tools/check_licences.py serve
```

`serve` and `mcp` are conflicting extras, so the second environment is a second run:

```sh
uv sync --extra mcp
uv run python -c "import mcp.server.context, seahaven.mcp"
uv run ty check src/seahaven/mcp tests/test_mcp_server.py tests/test_mcp_process.py
uv run pytest
uv run python tools/check_licences.py mcp
uv sync --extra serve                      # back to the everyday one
```

The type check is split between the two environments, because `ty` resolves imports against the one
it runs in. Each block carries its half: the first walks everything but the MCP paths, and the
second checks exactly those.

## Guidelines

- Everything is typed, tests included. `ty` is the checker.
- Write tests with the code. Examples in the docs are executed by the test suite.
- Keep a pull request to one change. Unrelated fixes go in their own.
- Runtime dependencies must be permissively licensed (MIT, Apache-2.0, BSD). No copyleft.
- No real customer data anywhere, including in the reference world.

## Licence

Seahaven is intended to be released under the MIT licence, but the `LICENSE` file is not in the
repository yet. Ask before investing in a large change.

>cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad && gh api repos/Kiln-AI/Seahaven/git/trees/main?recursive=1 --jq '.tree[].path' | grep -v '^worlds/.*/data' | head -300; echo ---; gh search code --repo Kiln-AI/Seahaven -- "fidelity" --json path -q '.[].path' 2>&1 | head; gh search code --repo Kiln-AI/Seahaven -- "accuracy" --json path -q '.[].path' 2>&1 | head; gh search code --repo Kiln-AI/Seahaven -- "realism" --json path -q '.[].path' 2>&1 | head

(eval):cd:1: no such file or directory: /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad
---
HTTP 422: ERROR_TYPE_QUERY_PARSING_FATAL unable to parse query! (https://api.github.com/search/code?page=1&per_page=30&q=fidelity+--json+path+-q+.%5B%5D.path+repo%3AKiln-AI%2FSeahaven)
HTTP 422: ERROR_TYPE_QUERY_PARSING_FATAL unable to parse query! (https://api.github.com/search/code?page=1&per_page=30&q=accuracy+--json+path+-q+.%5B%5D.path+repo%3AKiln-AI%2FSeahaven)
HTTP 422: ERROR_TYPE_QUERY_PARSING_FATAL unable to parse query! (https://api.github.com/search/code?page=1&per_page=30&q=realism+--json+path+-q+.%5B%5D.path+repo%3AKiln-AI%2FSeahaven)

Cloning the repo is simpler than searching through the API.

>mkdir -p /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src && cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src && gh repo clone Kiln-AI/Seahaven -- --depth 1 -q && cd Seahaven && ls && git ls-files | grep -vE '\.(json|jsonl|csv)$' | head -200 && echo --- && grep -rniE "fidelity|accura|realis|plausib|confidence|drift|grounded|judge" --include=*.py --include=*.md -l . | head -50

AGENTS.md
extensions
LICENSE
pyproject.toml
README.md
specs
src
tests
tools
uv.lock
worlds
.github/CONTRIBUTING.md
.github/RELEASING.md
.github/workflows/ci.yml
.gitignore
AGENTS.md
LICENSE
README.md
extensions/seahaven-xmlrpc/.gitignore
extensions/seahaven-xmlrpc/README.md
extensions/seahaven-xmlrpc/pyproject.toml
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/__init__.py
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/call_log.py
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/documents.py
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/faults.py
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/middleware.py
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/py.typed
extensions/seahaven-xmlrpc/src/seahaven_xmlrpc/tool.py
extensions/seahaven-xmlrpc/tests/conftest.py
extensions/seahaven-xmlrpc/tests/test_call_log.py
extensions/seahaven-xmlrpc/tests/test_contract.py
extensions/seahaven-xmlrpc/tests/test_dispatch.py
extensions/seahaven-xmlrpc/tests/test_faults.py
extensions/seahaven-xmlrpc/tests/test_registration.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/__init__.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/methods.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/middleware/__init__.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/middleware/error_handler.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/middleware/faults.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/startup.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/tools/__init__.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/tools/rpc.py
extensions/seahaven-xmlrpc/tests/tracker_rpc/world.py
pyproject.toml
specs/projects/add_mcp/architecture.md
specs/projects/add_mcp/control_tools_task.md
specs/projects/add_mcp/functional_spec.md
specs/projects/add_mcp/implementation_plan.md
specs/projects/add_mcp/phase_plans/phase_1.md
specs/projects/add_mcp/phase_plans/phase_2.md
specs/projects/add_mcp/phase_plans/phase_3.md
specs/projects/add_mcp/phase_plans/phase_4.md
specs/projects/add_mcp/project_overview.md
specs/projects/clock_modes/architecture.md
specs/projects/clock_modes/backlog.md
specs/projects/clock_modes/functional_spec.md
specs/projects/clock_modes/implementation_plan.md
specs/projects/clock_modes/phase_plans/phase_1.md
specs/projects/clock_modes/phase_plans/phase_2.md
specs/projects/clock_modes/phase_plans/phase_3.md
specs/projects/clock_modes/phase_plans/phase_4.md
specs/projects/clock_modes/project_overview.md
specs/projects/clock_modes/risk_report.md
specs/projects/cr_moderate_fixes/architecture.md
specs/projects/cr_moderate_fixes/components/phase_1_batch.md
specs/projects/cr_moderate_fixes/components/phase_2_bulk_gate.md
specs/projects/cr_moderate_fixes/components/phase_3_route_middleware.md
specs/projects/cr_moderate_fixes/components/phase_4_docs.md
specs/projects/cr_moderate_fixes/functional_spec.md
specs/projects/cr_moderate_fixes/implementation_plan.md
specs/projects/cr_moderate_fixes/phase_plans/phase_1.md
specs/projects/cr_moderate_fixes/phase_plans/phase_2.md
specs/projects/cr_moderate_fixes/phase_plans/phase_3.md
specs/projects/cr_moderate_fixes/phase_plans/phase_4.md
specs/projects/cr_moderate_fixes/project_overview.md
specs/projects/cr_moderate_fixes/triage.md
specs/projects/reset_schema/architecture.md
specs/projects/reset_schema/functional_spec.md
specs/projects/reset_schema/implementation_plan.md
specs/projects/reset_schema/phase_plans/phase_1.md
specs/projects/reset_schema/phase_plans/phase_2.md
specs/projects/reset_schema/project_overview.md
specs/projects/rest_apis/architecture.md
specs/projects/rest_apis/functional_spec.md
specs/projects/rest_apis/implementation_plan.md
specs/projects/rest_apis/phase_plans/phase_1.md
specs/projects/rest_apis/phase_plans/phase_2.md
specs/projects/rest_apis/phase_plans/phase_3.md
specs/projects/rest_apis/phase_plans/phase_4.md
specs/projects/rest_apis/project_overview.md
specs/projects/seahaven_framework/architecture.md
specs/projects/seahaven_framework/components/cli_and_check.md
specs/projects/seahaven_framework/components/fixtures_instances.md
specs/projects/seahaven_framework/components/helpers_and_control.md
specs/projects/seahaven_framework/components/openenv.md
specs/projects/seahaven_framework/components/projecttracker.md
specs/projects/seahaven_framework/components/pytest_and_docs.md
specs/projects/seahaven_framework/components/runtime_db.md
specs/projects/seahaven_framework/components/world_and_dispatch.md
specs/projects/seahaven_framework/functional_spec.md
specs/projects/seahaven_framework/implementation_plan.md
specs/projects/seahaven_framework/phase_plans/phase_1.md
specs/projects/seahaven_framework/phase_plans/phase_10.md
specs/projects/seahaven_framework/phase_plans/phase_11.md
specs/projects/seahaven_framework/phase_plans/phase_12.md
specs/projects/seahaven_framework/phase_plans/phase_13.md
specs/projects/seahaven_framework/phase_plans/phase_2.md
specs/projects/seahaven_framework/phase_plans/phase_3.md
specs/projects/seahaven_framework/phase_plans/phase_4.md
specs/projects/seahaven_framework/phase_plans/phase_5.md
specs/projects/seahaven_framework/phase_plans/phase_6.md
specs/projects/seahaven_framework/phase_plans/phase_7.md
specs/projects/seahaven_framework/phase_plans/phase_8.md
specs/projects/seahaven_framework/phase_plans/phase_9.md
specs/projects/seahaven_framework/project_overview.md
specs/projects/startup_sql/architecture.md
specs/projects/startup_sql/functional_spec.md
specs/projects/startup_sql/implementation_plan.md
specs/projects/startup_sql/phase_plans/phase_1.md
specs/projects/startup_sql/phase_plans/phase_2.md
specs/projects/startup_sql/phase_plans/phase_3.md
specs/projects/startup_sql/project_overview.md
specs/projects/state_v2/architecture.md
specs/projects/state_v2/functional_spec.md
specs/projects/state_v2/ideas_for_discussion.md
specs/projects/state_v2/implementation_plan.md
specs/projects/state_v2/phase_plans/phase_1.md
specs/projects/state_v2/phase_plans/phase_2.md
specs/projects/state_v2/phase_plans/phase_3.md
specs/projects/state_v2/phase_plans/phase_4.md
specs/projects/state_v2/phase_plans/phase_5.md
specs/projects/state_v2/phase_plans/phase_6.md
specs/projects/state_v2/project_overview.md
specs/projects/state_v2/research/state-formats-and-judges/benchmarks-with-stateful-backends/benchmark-profiles.md
specs/projects/state_v2/research/state-formats-and-judges/benchmarks-with-stateful-backends/cross-cutting-patterns.md
specs/projects/state_v2/research/state-formats-and-judges/benchmarks-with-stateful-backends/state-diff-dsl.md
specs/projects/state_v2/research/state-formats-and-judges/benchmarks-with-stateful-backends/summary.md
specs/projects/state_v2/research/state-formats-and-judges/declarative-assertion-languages-over-json/eval-tooling-assertion-shapes.md
specs/projects/state_v2/research/state-formats-and-judges/declarative-assertion-languages-over-json/hands-on-trials.md
specs/projects/state_v2/research/state-formats-and-judges/declarative-assertion-languages-over-json/judge-shape-recommendation.md
specs/projects/state_v2/research/state-formats-and-judges/declarative-assertion-languages-over-json/language-comparison.md
specs/projects/state_v2/research/state-formats-and-judges/declarative-assertion-languages-over-json/summary.md
specs/projects/state_v2/research/state-formats-and-judges/diff-formats-and-versioned-data-contracts/implications-for-state-format.md
specs/projects/state_v2/research/state-formats-and-judges/diff-formats-and-versioned-data-contracts/row-change-formats.md
specs/projects/state_v2/research/state-formats-and-judges/diff-formats-and-versioned-data-contracts/sqlite-session-and-sqldiff.md
specs/projects/state_v2/research/state-formats-and-judges/diff-formats-and-versioned-data-contracts/summary.md
specs/projects/state_v2/research/state-formats-and-judges/diff-formats-and-versioned-data-contracts/versioned-data-contracts.md
specs/projects/state_v2/research/state-formats-and-judges/openenv-and-rl-framework-consumers/inspect-ai-state-and-scorers.md
specs/projects/state_v2/research/state-formats-and-judges/openenv-and-rl-framework-consumers/openenv-state-and-rewards.md
specs/projects/state_v2/research/state-formats-and-judges/openenv-and-rl-framework-consumers/openenv-tasks-and-episode-persistence.md
specs/projects/state_v2/research/state-formats-and-judges/openenv-and-rl-framework-consumers/rl-trainer-consumers.md
specs/projects/state_v2/research/state-formats-and-judges/openenv-and-rl-framework-consumers/summary.md
specs/projects/state_v2/research/state-formats-and-judges/openenv-and-rl-framework-consumers/verifiers-and-prime-intellect.md
specs/projects/state_v2/research/state-formats-and-judges/research_plan.md
specs/projects/state_v2/research/state-formats-and-judges/summary.md
specs/projects/state_v2/research/state-formats-and-judges/tool-call-and-trajectory-records/eval-harness-records.md
specs/projects/state_v2/research/state-formats-and-judges/tool-call-and-trajectory-records/otel-genai-and-openinference.md
specs/projects/state_v2/research/state-formats-and-judges/tool-call-and-trajectory-records/recommendation-for-seahaven.md
specs/projects/state_v2/research/state-formats-and-judges/tool-call-and-trajectory-records/summary.md
specs/projects/state_v2/research/state-formats-and-judges/tool-call-and-trajectory-records/trajectory-container-formats.md
specs/projects/state_v2/research/state-formats-and-judges/tool-call-and-trajectory-records/wire-formats-and-action-lists.md
specs/projects/world_composition/architecture.md
specs/projects/world_composition/backlog.md
specs/projects/world_composition/functional_spec.md
specs/projects/world_composition/implementation_plan.md
specs/projects/world_composition/phase_plans/phase_1.md
specs/projects/world_composition/phase_plans/phase_2.md
specs/projects/world_composition/phase_plans/phase_3.md
specs/projects/world_composition/phase_plans/phase_4.md
specs/projects/world_composition/phase_plans/phase_5.md
specs/projects/world_composition/phase_plans/phase_6.md
specs/projects/world_composition/phase_plans/phase_7.md
specs/projects/world_composition/project_overview.md
src/.gitattributes
src/seahaven/__init__.py
src/seahaven/call.py
src/seahaven/changes.py
src/seahaven/cli/__init__.py
src/seahaven/cli/check.py
src/seahaven/cli/docs.py
src/seahaven/cli/fixture.py
src/seahaven/cli/hub.py
src/seahaven/cli/mcp.py
src/seahaven/cli/new.py
src/seahaven/cli/scaffold.py
src/seahaven/cli/serve.py
src/seahaven/cli/templates/base/.gitignore.tmpl
src/seahaven/cli/templates/base/AGENTS.md.tmpl
src/seahaven/cli/templates/base/README.md.tmpl
src/seahaven/cli/templates/base/fixtures/.gitkeep.tmpl
src/seahaven/cli/templates/base/fixtures_src/generate.py.tmpl
src/seahaven/cli/templates/base/pyproject.toml.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/__init__.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/errors.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/middleware/__init__.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/middleware/error_handler.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/openenv_app.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/schema/001_items.sql.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/tools/__init__.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/tools/items.py.tmpl
src/seahaven/cli/templates/base/src/PACKAGE/world.py.tmpl
src/seahaven/cli/templates/base/tests/test_fixtures.py.tmpl
src/seahaven/cli/templates/base/tests/test_items.py.tmpl
src/seahaven/cli/templates/hub/Dockerfile.tmpl
src/seahaven/cli/templates/hub/__init__.py.tmpl
src/seahaven/cli/templates/hub/client.py.tmpl
src/seahaven/cli/templates/hub/models.py.tmpl
src/seahaven/cli/templates/hub/openenv.yaml.tmpl
src/seahaven/cli/templates/hub_readme/connecting.md.tmpl
src/seahaven/cli/templates/hub_readme/space_card.md.tmpl
src/seahaven/clock.py
---
(eval):1: no matches found: --include=*.py
Shell cwd was reset to /Users/karamelton

No content

>cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src/Seahaven && sed -n 1,200p README.md; echo ======; sed -n 1,80p src/seahaven/cli/check.py; echo =====; grep -rniE "fidelity|accura|realis|plausib|confidence|how (real|close)" src worlds README.md AGENTS.md src/seahaven/cli/templates | head -40

Exit code 1
<p align="center">
  <img width="200" height="168" alt="seahaven logo" src="https://github.com/user-attachments/assets/79e3b5cd-3522-48ae-b5a6-ba356da4ee00" />
</p>
<h3 align="center">
  Synthetic world framework for agent evals and RL.
</h3>

<p align="center">
  <a href="#quickstart"><strong>Quick Start</strong></a> •
  <a href="src/seahaven/docs/index.md"><strong>Docs</strong></a> •
  <a href="#example-worlds"><strong>Examples</strong></a>
</p>

<p align="center">
  <a href="https://pypi.org/project/seahaven/"><img src="https://img.shields.io/pypi/v/seahaven?logo=pypi&label=PyPI&logoColor=gold" alt="PyPI"></a>
  <a href="https://github.com/Kiln-AI/Seahaven/actions/workflows/ci.yml"><img src="https://github.com/Kiln-AI/Seahaven/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-brightgreen" alt="MIT License"></a>
</p>

Evals and RL need thousands of agent runs, each isolated, starting from a known state, and graded
on what the agent changed. Production systems can't do that. Seahaven is a Python framework for
building synthetic worlds that can: working copies of your agent's tools, realistic enough that the
agent can't tell the difference.

Seahaven handles the hard parts: parallel instances, reproducibility, serving, and change logs. You only write what's specific to your world: its tables and its tools.

<p align="center">
  <a href="https://www.youtube.com/watch?v=E1l-EKsO0Q8">
    <img width="300" alt="Watch the Seahaven video" src="https://github.com/user-attachments/assets/3c0bfbf7-1c4a-45fd-8c0b-a1a467d130d8" />
  </a>
</p>

> Named after the town in *The Truman Show*: an entire world built so that one inhabitant believes
> it is real.

## Features

### Realistic Worlds

- **[Recreate Any Environment](src/seahaven/docs/authoring.md#writing-a-tool):** Mock AI tool
  calls, REST APIs, sandboxed SQL, search, or any custom format.
- **[Stateful](src/seahaven/docs/concepts.md#instance):** Each instance of a world has its own independent
  SQLite database.
- **[Composable](#composing-worlds):** Compose, reuse and share worlds. Example: MyCoWorld
  can include [StripeAPIWorld](https://github.com/Kiln-AI/stripe_world) and ShopifyAPIWorld.

### Built for Evals and RL

- **[Fixtures](src/seahaven/docs/db_schema_and_fixtures.md):** Freeze known starting states like
  `small_startup`, `agency` or `big_co`, and reuse them across runs.
- **[Concurrent Instances](src/seahaven/docs/serving_and_openenv.md):** Serve hundreds of world
  instances per process, at thousands of requests per second.
- **[Evaluate World State](src/seahaven/docs/state.md):** Grade on state, not on transcripts.
  Every row the agent changed is logged.
- **[Reproducible](src/seahaven/docs/concepts.md#reproducibility):** Same initial state (fixture),
  same clock/time, same random seed: the same run, every time.

### Connect Anything

- **[OpenEnv](#serve-with-openenv):** `seahaven serve` is an OpenEnv environment. Drive it with any
  OpenEnv client, in any language, or publish it to Hugging Face.
- **[Web Console](src/seahaven/docs/serving_and_openenv.md#the-web-console):** `seahaven serve`
  includes a web UI: open instances, call tools, and inspect state in your browser.
- **[MCP](#use-with-mcp-clients):** `seahaven mcp` serves one world to an MCP client, so you can
  work against it by hand from an editor or chat app.

### Easy to Build

- **[Built for Coding Agents](#build-worlds-with-your-coding-agent):** Docs optimized for agents
  authoring worlds. `seahaven check` tells an agent the exact fix for every mistake.
- **[Just Python](src/seahaven/docs/authoring.md#writing-a-tool):** Tools are just functions.
  Tests use pytest. Your agent already knows how to write and test Seahaven worlds.

## Seahaven vs. Real Systems and Mocks

|                                      | Seahaven | Production or staging | Hand-written mocks |
|--------------------------------------|:--------:|:---------------------:|:------------------:|
| Realistic tools and data             |    ✅    |          ✅           |         ❌         |
| Stateful across arbitrary tool calls |    ✅    |          ✅           |         ❌         |
| A private instance for every run     |    ✅    |          ❌           |         ✅         |
| Hundreds of parallel instances       |    ✅    |          ❌           |         ✅         |
| Every run starts from a known state  |    ✅    |          ❌           |         ✅         |
| Reproducible                         |    ✅    |          ❌           |         ✅         |
| Every change logged for grading      |    ✅    |          ❌           |         ❌         |
| Safe for the agent to break things   |    ✅    |          ❌           |         ✅         |

## Quickstart

**Create a world.** This writes a complete project: schema, tools, tests, a fixture generator, and
an `AGENTS.md` that points your coding agent at the docs.

```sh
uvx seahaven new crm_world # your world name
cd crm_world && uv sync
```

**Write your world.** A world is a schema and a set of tools. Here is a small CRM:

```python
import seahaven

world = seahaven.World(
    name="crm",
    version="1.0.0",
    schema="""
    CREATE TABLE contacts (
        id TEXT PRIMARY KEY,
        email TEXT NOT NULL,
        stage TEXT NOT NULL,
        updated_at TEXT NOT NULL
    ) STRICT;
    """,
    state_format="seahaven.state/1",
)


@world.tool
def create_lead(ctx: seahaven.Ctx, email: str) -> dict[str, str]:
    """Add a contact to the pipeline as a new lead."""
    lead = {"id": ctx.ids.uuid(), "email": email, "stage": "lead", "updated_at": ctx.clock.iso()}
    ctx.db.execute("INSERT INTO contacts VALUES (?, ?, ?, ?)", *lead.values())
    return lead


@world.tool
def list_stale_leads(ctx: seahaven.Ctx) -> list[dict[str, object]]:
    """List leads nobody has touched in 30 days."""
    return ctx.db.rows(
        "SELECT * FROM contacts WHERE stage = 'lead' "
        "AND updated_at < strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-30 days')"
    )
```

**Freeze a starting state.** A fixture is a frozen database that every run starts from:

```py
with world.instance(now="2026-06-01T09:00:00.000Z", clock_mode="fixed") as inst:
    for n in range(500):
        inst.call("create_lead", email=f"lead{n}@example.com")
    inst.freeze("big_co", "A pipeline of 500 new leads.")
```

**Run your agent.** Each run gets a private copy of the fixture in milliseconds. The same seed
replays the same run, and what the agent changed is a document you grade:

```py
for rollout in range(100):
    with world.instance("big_co", seed=rollout) as inst:
        run_agent(inst)  # your agent, your harness
        reward = grade(inst.state())  # every row the agent changed
```

**Serve it.** `seahaven serve` hosts an OpenEnv endpoint where every connection gets its own
instance. Open `http://127.0.0.1:8000/console` to drive it by hand.

```sh
uv run --extra serve seahaven serve
```

```py
from seahaven.openenv import SeahavenClient

with SeahavenClient(base_url="http://127.0.0.1:8000") as env:
    env.reset(fixture="big_co", seed=42)
    env.call("create_lead", email="[REDACTED]")
    final_state = env.state()  # the document the eval grades
```

## Example Worlds

- **[ProjectTracker](worlds/projecttracker/)**: the reference world, a fictional issue tracker
  shaped like Linear or Jira. Nine tables, 25 tools, full-text search, and fixtures from an empty
  workspace to a twelve-person agency with six months of history. Start here to learn the patterns
  ([walkthrough](src/seahaven/docs/projecttracker.md)).
- **[Stripe World](https://github.com/Kiln-AI/stripe_world)**: a mock of Stripe's Billing and
  Payments core, with 24 tables and 155 API operations behind the same tools as Stripe's own MCP
  server. It also serves Stripe's REST API, so the Stripe SDKs work against it unchanged.

## Composing Worlds

Build a world once and reuse it everywhere. A company world can add a payments world, such as
[Stripe World](https://github.com/Kiln-AI/stripe_world), and a chat world, plus its own tables and
tools. The agent sees one tool list, and an eval grades what changed in every world from one state
document. See the [composition docs](src/seahaven/docs/composition.md).

```py
company.add_world(payments_world.world, name="payments", tool_prefix="pay_")
company.add_world(chat_world.world, name="chat", tool_prefix="chat_")


@company.tool
def refund_order(ctx: seahaven.Ctx, charge_id: str, channel: str) -> dict[str, object]:
    """Refund a charge and tell the support channel it is done."""
    refund = ctx.worlds.payments.call("create_refund", charge_id=charge_id)
    ctx.worlds.chat.call("post_message", channel=channel, text=f"refunded {refund['amount']}")
    return refund
```

## Serve with OpenEnv

(eval):1: ===== not found

>cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src/Seahaven && sed -n 1,60p src/seahaven/cli/check.py; echo "-----"; grep -rniE "fidelity|accura|realis|plausib|how (real|close)|indistinguish" src worlds AGENTS.md | grep -v "^specs" | head -40; echo "-----"; ls src/seahaven/docs

"""`seahaven check`: every framework rule as a lint with a code and a named fix.

The whole command is discovery, `lint.run_all`, and printing. What makes it worth
a module of its own is the one case discovery cannot hand to the lints: a world
that could not be imported has no `World` to lint, and the failure is itself a
finding -- SH501 for a package with no world in it, SH104 for one whose DDL
SQLite refused. Reported as a line like any other, never as a traceback, because
an authoring agent reads the first and re-runs on the second.

Paths are printed relative to the *project*, not to wherever the shell happened
to be: `seahaven check` from `src/mypkg/middleware/` finds the same world as
`seahaven check` from the project root, and the two should not describe the same
file differently.
"""

import argparse
from dataclasses import dataclass
from pathlib import Path

from seahaven import lint
from seahaven.cli import CliError, add_world_option, discover
from seahaven.lint import Finding
from seahaven.lint.ddl import ddl_execution_finding

__all__ = ["Report", "add_parser", "collect", "run"]


@dataclass(frozen=True)
class Report:
    """What a check found, and the project the paths in it are relative to."""

    root: Path
    findings: list[Finding]

    @property
    def failed(self) -> bool:
        """Whether anything found is an error. A warning alone does not stop a commit."""
        return any(finding.severity == "error" for finding in self.findings)


def add_parser(subcommands: argparse._SubParsersAction[argparse.ArgumentParser]) -> None:
    parser = subcommands.add_parser(
        "check",
        help="run every lint over this world",
        description="Run every lint over this world.",
    )
    add_world_option(parser)
    parser.set_defaults(handler=run)


def run(args: argparse.Namespace) -> int:
    """Print every finding, sorted; 1 if any of them is an error."""
    report = collect(args.world, Path.cwd())
    for finding in report.findings:
        print(finding.render(report.root))
    return 1 if report.failed else 0


def collect(explicit: str | None, start: Path) -> Report:
    """Every finding for the world `start` resolves to, and that world's root.
-----
src/seahaven/openenv/__init__.py:134:        "to be thrown away -- with a 200 and a plausible body. That is an upstream defect in "
src/seahaven/fixtures.py:332:    indistinguishable from an ordinary file here. It is the weaker variant --
src/seahaven/lint/code.py:25:prefix or accepts the infidelity.
src/seahaven/docs/authoring.md:592:left visible that the real integration never exposes, is your infidelity rather than the added
src/seahaven/docs/composition.md:186:infidelity. Those are the two answers.
src/seahaven/docs/http_apis.md:10:  principles behind Seahaven, such as many isolated and realistic instances, help in software
worlds/projecttracker/tests/test_pagination.py:85:    """The whole point of the third field: the page would otherwise be plausible and wrong."""
worlds/projecttracker/tests/test_sql_tools.py:132:    # rows would still have answered with three plausible numbers.
worlds/projecttracker/src/projecttracker/tools/_pagination.py:22:ordering and the wrong page would look plausible.
-----
authoring.md
clock.md
composition.md
concepts.md
db_schema_and_fixtures.md
extensions.md
http_apis.md
index.md
projecttracker.md
reference
serving_and_openenv.md
state.md
testing.md
Shell cwd was reset to /Users/karamelton

>cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src/Seahaven && grep -rniE "contract|real (api|system|service)|record(ed)? (from|against)|golden" src/seahaven/docs | head -30; echo ----; sed -n 1,40p extensions/seahaven-xmlrpc/tests/test_contract.py; echo ----; python3 -c "
import re,pathlib
for p in pathlib.Path('src/seahaven/cli').glob('*.py'):
  for m in re.finditer(r'add_parser\(\s*\"([a-z-]+)\".*?help=\"([^\"]+)\"', p.read_text(), re.S): print(m.group(1),'-',m.group(2))
"

src/seahaven/docs/extensions.md:126:whole contract is that failures are fault documents would answer with something else.
src/seahaven/docs/testing.md:101:is the contract; the wording is presentation.
src/seahaven/docs/state.md:37:| [The compatibility contract](#the-compatibility-contract) | What a reader may rely on for as long as a document exists |
src/seahaven/docs/state.md:75:| `world` | `{name, version}` of the **root** world. The version is the world's contract: a schema, tool or format change bumps it |
src/seahaven/docs/state.md:96:[compatibility contract](#the-compatibility-contract) is what that promise is worth.
src/seahaven/docs/state.md:260:  `WorldBug`, which is the formatter author's contract to keep.
src/seahaven/docs/state.md:534:## The compatibility contract
src/seahaven/docs/state.md:536:Two contracts, one per half of the document.
src/seahaven/docs/state.md:551:`world.version` is the root world's contract and `composition[path]["world_version"]` each node's:
src/seahaven/docs/index.md:9:run in parallel with the others, and be inspectable afterwards. A real system cannot do that, and
src/seahaven/docs/composition.md:324:  what two real services do, and Seahaven does not paper over it. Eval authors writing state checks
----
"""The contract of `functional_spec.md` §21, asserted rather than asserted-to.

This package exists to prove that an extension can carry a protocol Seahaven has
never heard of, over the published seams and nothing else. The tests in the other
modules prove the protocol works; these prove it was built the way the contract
says, because an example that quietly reached past the contract would teach the
wrong thing to everyone who copied it.
"""

import ast
import importlib
import importlib.metadata
import sys
from pathlib import Path

import pytest

import seahaven
import seahaven_xmlrpc
from seahaven.cli import check
from seahaven.world import CONTROL_TOOL_NAMES

_EXTENSION = Path(seahaven_xmlrpc.__file__).parent
_FRAMEWORK = Path(seahaven.__file__).parent
_WORLD = "tracker_rpc:world"


def _modules(root: Path) -> list[Path]:
    return sorted(root.rglob("*.py"))


def test_importing_the_extension_registers_nothing() -> None:
    """An extension must not register itself: the world registers what it wants.

    A `World` built after this package is imported carries the framework's own two
    control tools and nothing else. Nothing in the extension runs at import: no
    tool, no middleware, no startup hook, no schema.
    """
    world = seahaven.World(
        "empty_after_import",
----
fixture - list, freeze and fork this world's fixtures
list - every fixture: id, parent, now, description
freeze - fill a blank instance and freeze it
fork - fill an instance of a fixture and freeze the result
check - run every lint over this world
docs - print the directory holding the bundled docs
serve - run this world's OpenEnv server
hub - add the files a world needs to publish to a hub
mcp - serve this world to an MCP client over stdio
new - scaffold a new world
Shell cwd was reset to /Users/karamelton

No. Neither CONTRIBUTING.md nor any Seahaven command estimates how accurate the simulated world is, and nothing asks an AI to judge it either.

What CONTRIBUTING.md covers: it's only for people working on the framework itself. It explains the uv/Python 3.14 setup, the CI checks they must pass (ruff, ty, pytest, the licence check), and how to work with the two conflicting environments (serve and mcp). It doesn't describe anything for world users.

The full command list (I cloned the repo and read src/seahaven/cli/):

Command	What it does
`new`	Creates a new world project
`check`	Runs every lint over the world
`fixture list / freeze / fork`	Manages saved starting states
`serve`	Runs the OpenEnv server and web console
`mcp`	Serves the world to an MCP client
`hub`	Adds the files needed to publish to a hub
`docs`	Prints the path to the bundled docs

Why check doesn't do this: it sounds like the closest match, but it tests whether the world follows Seahaven's own rules. Examples are bad SQL in the schema or a package with no world in it, and each finding comes with a code and a fix. It never compares the world to the real system it's copying.

How the project treats accuracy: it's left to whoever builds the world. The README promises worlds "realistic enough that the agent can't tell the difference," but there's no way to measure that. The docs only mention "infidelity" as a tradeoff the author makes knowingly (docs/authoring.md:592, docs/composition.md:186).

What you can do instead:

Check it by hand: seahaven serve opens a web console at /console where you can call tools and look at the data. seahaven mcp lets you poke at the world from Claude or an editor.
Get an AI's opinion: point a coding agent at the world's tools and schema plus the real API's docs, and ask it to list where they differ: missing fields, error formats, pagination, edge cases. That's the closest thing to the "AI's take" you're after, but you'd have to set it up yourself.
Ask the maintainers: CONTRIBUTING asks people to open an issue before starting anything bigger than a bug fix. A "fidelity report" command would count, so an issue is the place to propose one.

No content

Help me understand how this solution would be built?

>cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src/Seahaven && sed -n 1,200p src/seahaven/cli/__init__.py | grep -nE "def |import|add_parser|SUBCOMMANDS|check|serve" | head -40; echo ----; ls src/seahaven/lint; sed -n 1,80p src/seahaven/lint/__init__.py | grep -nE "class|def |code|severity|fix" | head -30; echo ----; grep -nE "def (tools|tool|instance|call|schema|describe)|self\.(tools|_tools)\b|class World" src/seahaven/world.py | head -30

5:already exists -- `World.instance`, `Instance.freeze`, `seahaven.openenv.serve`,
19:`import_module` says only that Python was involved. Exit 2 is argparse's, for a
28:import argparse
29:import re
30:import sys
31:import tomllib
32:import traceback
33:from dataclasses import dataclass
34:from importlib import import_module
35:from pathlib import Path
36:from types import ModuleType
38:from seahaven.errors import SeahavenError
39:from seahaven.world import DDL_DOES_NOT_EXECUTE, World
63:# Seahaven's own package: its frames are never where a world's import failed.
70:    `code` is set when `seahaven check` has to render the failure as a finding
71:    rather than as a user error: an import that never produced a `World` is
76:    `path` and `line` place a finding, `fix` is the edit `check` prints beside
77:    it, and `root` is the project `check` shows the path relative to: the same
81:    def __init__(
106:    # `sys.modules` immediately after the package was imported: what SH301 has to
108:    imported: frozenset[str]
112:def main(argv: list[str] | None = None) -> int:
122:def build_parser() -> argparse.ArgumentParser:
129:    # Imported here rather than at module top: every subcommand module imports
131:    from seahaven.cli import check, docs, fixture, hub, mcp, new, serve
135:    for module in (new, hub, check, docs, fixture, serve, mcp):
136:        module.add_parser(subcommands)
140:def add_world_option(parser: argparse.ArgumentParser) -> None:
147:            "the world to act on, as an importable module and the attribute holding it; "
153:def check_world_option(explicit: str | None) -> None:
154:    """Refuse a `--world` that is not `module:attr`, without importing anything.
156:    `discover` asks the same question on its way to the import, and every
158:    the server, so that a failed import reaches the MCP client as an error
161:    anything is served (MCP functional spec §2.4).
167:def _split_world_option(explicit: str) -> tuple[str, str]:
177:def find_world(explicit: str | None, start: Path | None = None) -> World:
182:def discover(explicit: str | None, start: Path | None = None) -> Discovery:
183:    """`find_world`, plus the module, the project root and the import snapshot."""
187:        _make_importable(start, start / SOURCE_DIRNAME)
188:        return _import(module_name, attribute, root=start)
----
__init__.py
code.py
coverage.py
ddl.py
fixtures.py
world.py
6:concatenation of the five rule modules in code order. `seahaven/cli/check.py` is
9:The codes are stable and their gaps are deliberate: a retired rule's number is
10:never reused, so a fix written against `SH203` in a world's history always means
19:| SH201 | warning | a wall-clock call in world code, outside `middleware/` |
20:| SH203 | warning | `random` or `uuid.uuid4()` in world code |
22:| SH206 | warning | a prefixed world's description names a sibling tool the agent cannot call |
27:| SH401 | error | a fixture sidecar that does not validate |
28:| SH402 | error | a fixture's `file_sha256` does not match its state file |
29:| SH403 | error | a fixture's `schema_hash` does not match the world |
30:| SH404 | error | a fixture's `now` is not canonical |
31:| SH405 | error | a fixture's state file has `-wal` or `-shm` companions |
34:| SH502 | error | a `Worlds` subclass annotates a name no `add_world` registered |
35:| SH503 | warning | a registered child name no declared `Worlds` subclass annotates |
44:from dataclasses import dataclass
58:@dataclass(frozen=True)
59:class Finding:
62:    `fix` is not optional and is not prose about the problem: every rule here
67:    code: str
68:    severity: Severity
71:    fix: str
72:    # A fixture's sidecar, a module that was never imported: some findings are
77:    def render(self, root: Path | None = None) -> str:
----
118:class World:
190:        self._tools: dict[str, Tool] = {}
224:    def tools(self) -> Mapping[str, Tool]:
226:        return MappingProxyType(self._tools)
229:    def tools_by_fn(self) -> Mapping[Callable[..., Any], tuple[Tool, ...]]:
294:    def tool[**P, R](self, obj: Tool[P, R], /) -> Tool[P, R]: ...
296:    def tool[**P, R](
306:    def tool[**P, R](
316:    def tool(
343:    def instance_startup(self, obj: StartupHook | None = None, /) -> Any:
515:    def instance(
596:        twin._tools = dict(self._tools)
664:        if tool.name in self._tools:
669:        self._tools[tool.name] = tool
Shell cwd was reset to /Users/karamelton

>cd /private/tmp/claude-501/-Users-karamelton/4bd88558-132e-470e-aa92-7b22660bc345/scratchpad/src/Seahaven && grep -nE "^class Tool|^    (name|description|input_schema|output_schema|schema|params)\b.*:" src/seahaven/*.py | head -20; grep -n "def state\|def call\|def changes" src/seahaven/instance.py src/seahaven/*.py 2>/dev/null | head

src/seahaven/call.py:58:    name: str
src/seahaven/composition.py:138:    name: str
src/seahaven/composition.py:224:    name: str
src/seahaven/composition.py:585:    name: str
src/seahaven/composition.py:609:    name: str
src/seahaven/fixtures.py:162:    description: str
src/seahaven/errors.py:41:class ToolErrorType(StrEnum):
src/seahaven/errors.py:67:class ToolError(SeahavenError):
src/seahaven/sandbox.py:390:    params: Sequence[SqlValue] = (),
src/seahaven/sandbox.py:523:    params: Sequence[SqlValue],
src/seahaven/tool.py:59:class Tool[**P = ..., R = Any]:
src/seahaven/tool.py:75:    name: str
src/seahaven/tool.py:76:    description: str
src/seahaven/tool.py:78:    params: type[pydantic.BaseModel]
src/seahaven/tool.py:79:    schema: dict[str, Any]
src/seahaven/instances.py:282:    name: str
src/seahaven/setup_sql.py:78:    description: str
src/seahaven/call.py:131:    def call_middleware(ctx: Ctx, call: Call) -> Any:
src/seahaven/clock.py:237:    def statement_started(self, event: dict[str, Any]) -> None:
src/seahaven/composition.py:528:    def call_innermost(ctx: Ctx[Any], call: Call) -> Any:
src/seahaven/fixtures.py:207:    def state_path(self) -> Path:
src/seahaven/handles.py:139:    def state(self) -> dict[str, Any]:
src/seahaven/handles.py:149:    def call[**P, R](
src/seahaven/handles.py:157:    def call(self, tool: str, /, **arguments: Any) -> Any: ...
src/seahaven/handles.py:159:    def call(self, tool: str | Callable[..., Any], /, *args: Any, **arguments: Any) -> Any:
src/seahaven/instances.py:398:    def state_path(self) -> Path:
src/seahaven/instances.py:403:    def call_count(self) -> int:
Shell cwd was reset to /Users/karamelton

A fidelity command would compare the world against a reference for the real system. Seahaven already gives you the world's side through code you can inspect. The work is getting the real system's side and deciding how to compare the two.

What the code already gives you

world.tools (src/seahaven/world.py:224) maps each tool name to a Tool. Each Tool has a name, a description, and a schema, which is the JSON Schema of its parameters, built from a pydantic model (src/seahaven/tool.py:75-79).
world.instance(fixture) with .call() and .state() (src/seahaven/handles.py) runs real tool calls against a throwaway copy of the world and shows exactly what changed.
discover() (src/seahaven/cli/__init__.py:182) finds and imports the world, the same way check does.
Finding (src/seahaven/lint/__init__.py:59) is the output format check uses: a code, a severity, a location and a fix. A fidelity command could reuse it, so its output would read like check's.

Three checks, from cheapest to most expensive

1. Compare the tool list (no AI needed). Take the world's tools and compare them to a reference: an OpenAPI spec, or the tools/list output from the real service's MCP server. Report:

tools that are missing or extra
parameters that are missing, renamed, or a different type
differences in which fields are required

The results are exact and repeatable, and they give you an honest number: "142 of 155 operations match the reference." Stripe World says it exposes "the same tools as Stripe's own MCP server," so it's a ready-made test case.

2. Compare behavior. Make the same calls against the world and against the real system's sandbox (for example, Stripe test mode), then compare:

the shape of the responses
error types and status codes
pagination
edge cases, like an ID that doesn't exist or a duplicate create

Compare shapes, not values, because the two have different data. Record the real responses once and save them, so later runs work offline and give the same results every time. That matters to this project: the README lists reproducibility as a main feature.

3. Ask an AI judge (the "AI's take" you mentioned). Some things can't be checked mechanically:

Is the fixture data realistic?
Are the error messages worded like the real ones?
Is behavior that's described in prose docs, but not in any spec, actually implemented?

Give a model the real system's docs, the world's tool descriptions and schemas, and sample outputs from check 2. Ask for structured findings, each one quoting the docs passage and the world output it's based on. Treat the result as an opinion. Requiring a quote for every finding is what keeps it from making things up. I wouldn't report a single "87% accurate" score here, because the number would look precise without meaning much. Per-tool numbers from checks 1 and 2 are defensible.

Where it fits in the code

As a core command, it would be a new src/seahaven/cli/fidelity.py with an add_parser function, added to the tuple at src/seahaven/cli/__init__.py:135, using discover() the way check does.

But checks 2 and 3 need network access, real-service credentials and model API keys. check deliberately doesn't (it's offline and gives the same output every time), and new dependencies have to be permissively licensed. So it probably fits better as a separate extension package, like extensions/seahaven-xmlrpc. You could also build only check 1 into core and keep the rest outside.

Suggested first version

Build only check 1:

It would load the world, load the reference tool list, compare names and parameter schemas, and print the differences in the same format as check. That's about a day's work, it needs no API keys, and it tests whether the idea is worth pursuing. Bring it to the maintainers in an issue before opening a PR, since CONTRIBUTING asks for that on anything bigger than a bug fix.

I can prototype check 1 against Stripe World if you'd like. Just tell me where you want the files. I'd suggest somewhere outside the repo, like your Desktop.

No content

So: in short -- there's no way, currently, to help the user reason about the fidelity/accuracy of the world that's been spun up?

That's right. Nothing built in measures or estimates how faithful a world is to the real system, and nothing asks an AI to judge it. I went through all the CLI commands and searched the docs and source, and found nothing that does this.

What you have is ways to inspect the world yourself, not ways to judge it:

seahaven serve runs a web console where you can call tools and look at the data.
seahaven mcp lets you explore the world from Claude or an editor.
inst.state() shows exactly what each call changed.
The world's own pytest tests show what the author chose to check.

So you can get a feel for the world by using it, but the comparison to the real system happens in your head, or in tests the author wrote. Seahaven doesn't produce any report or score for it.

No content